From 854b9a69f07871edb56920e1f51651226d50237a Mon Sep 17 00:00:00 2001 From: Phil Merrell Date: Mon, 20 Jul 2026 14:03:05 -0600 Subject: [PATCH] Release/1.9.0 (#707) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs(memory): reframe spec as Memory Spaces — bindable, templated, shareable Reframes the per-user markdown memory spec into the Memory Space primitive: a named, first-class, bindable, templated, and shareable markdown wiki. Chief-of-Staff template + a bound agent rather than a special-cased feature. - Adds the three-layer abstraction (Agent / Memory Space / declarative binding) and the structural-config vs. semantic-MEMORY.md split. - Space-keyed storage (SPACE#{id} + membership records + UserSpacesIndex GSI) so sharing is expressible from day one. - Entry types (entity / episodic / fact), Space Templates, and manifest- indexed fields for relational/temporal queries ("who owes what"). - Sharing via owner/editor/viewer grants mirroring assistant collab-edit, with run-as-user write attribution. - Data governance is proportionate: same data class as sessions/artifacts behind the same Entra JWT + RBAC — identity-based, no content-inspection gate. Deletion-purge + inherited encryption are the only real items. - 8-PR phasing; PR-1 (data layer) is the next buildout phase. Co-Authored-By: Claude Opus 4.8 * docs(kaizen): recover resourceOauth2ReturnUrl shape section into harness findings PR #576 merged from a state prior to commit 908f7e18, orphaning the resourceOauth2ReturnUrl parameter-shape + harness-security cross-check subsection. This re-applies it onto develop. Co-Authored-By: Claude Opus 4.8 * docs(memory): bake in full-ownership zip export of a Memory Space Add a first-class "download the entire space as a .zip of raw markdown" capability (index + all entries, structure preserved, metadata.json), framed as the user-ownership / zero-lock-in property. Promotes the stubbed export endpoint into §9 with contents, access, streaming mechanics, and an import-friendly round-trip note; folds the zip export into PR-5. Co-Authored-By: Claude Opus 4.8 * feat(memory): Memory Spaces data layer (PR-1) Adds the F5 Memory Space primitive's data layer — no runtime wiring, gated by MEMORY_SPACES_ENABLED (default off). Backend (apis/shared/memory/): - store.py: S3 content-addressed byte store (sibling of the skills store) - models.py: MemorySpace / MemoryIndex / MemoryEntryRef / SpaceMember - templates.py: Blank / Chief-of-Staff / Research-Notebook presets - repository.py: dedicated memory-spaces table CRUD (META/INDEX/MEMBER rows, OwnerIndex + MemberIndex GSIs, Decimal handling) - service.py: permission-gated lifecycle + sharing + entry/index I/O, resolve_permission chokepoint (viewer reads, editor writes, owner shares/deletes), content-addressed writes with GC-on-replace - feature_flags.py: memory_spaces_enabled() (default off) - 47 moto-backed tests; import boundaries clean Infrastructure: - MemorySpacesConstruct: S3 bucket + dedicated memory-spaces DynamoDB table (OwnerIndex/MemberIndex GSIs), a per-domain table matching the project's actual pattern - Threaded via PlatformComputeRefs to both compute roles (readwrite S3 + DynamoDB); env vars S3_MEMORY_SPACES_BUCKET_NAME / DYNAMODB_MEMORY_SPACES_TABLE_NAME / MEMORY_SPACES_ENABLED - config.ts flag default-off; resource-count assertions updated - tsc + 429 jest tests pass Spec updated to reflect the dedicated-table + two-GSI decisions. Co-Authored-By: Claude Opus 4.8 * docs(memory): re-slice phasing into primitive vs. agent-consumption workstreams Splits the plan into two workstreams to keep the Memory Space a clean bindable primitive: Workstream A (this epic) delivers the primitive + the user-facing "own your data" surface (data layer, app-api CRUD, export, sharing, SPA panel, consolidation); Workstream B (Agent/Harness layer) delivers agent-consumption — the memory_* tools, declarative binding, and system-prompt index injection — so any run surface can bind the same primitive rather than welding it to inference-api. Co-Authored-By: Claude Opus 4.8 * feat(memory): Memory Spaces user surface — /memory/spaces CRUD (A2) Workstream A2 of the re-sliced memory epic: the user-facing "own your data" surface over the Memory Space primitive. No agent-consumption (tools / binding / prompt injection) — that's the Agent/Harness workstream. app-api (apis/app_api/memory_spaces/): - routes.py: /memory/spaces CRUD over MemorySpaceService — list (with templates + accurate per-space role), create-from-template, get (index + entry manifest), delete-or-leave, entry read/list/upsert/delete, index read/update. Sync handlers (FastAPI threadpools the sync boto3 service). - Gated by require_memory_spaces_user: 404 while MEMORY_SPACES_ENABLED off (surface behaves as unmounted); cookie auth via get_current_user_from_session. - Service errors translated NotFound->404, Permission->403, Error->400. - models.py: camelCase request/response models. - Mounted before the existing /memory (AgentCore Memory) router; paths are non-overlapping (/memory/spaces vs /memory/{record_id}). shared service: - leave_space(): a member drops their own grant (the shared-in forget-me case); owner cannot leave. - list_spaces_for_user() now returns (space, role) so shared-in spaces carry the member's real viewer/editor grant (only consumer is the new route). Tests: 12 route tests (moto-backed real service, flag gate, CRUD, 403/404, member-leaves-via-delete) + leave_space service tests. Full memory suite + import boundaries green (69 passing). Co-Authored-By: Claude Opus 4.8 * feat(memory): Memory Space zip export — /memory/spaces/{id}/export (A3) The "own your data" leg of Workstream A: a loss-free `.zip` download of a space's raw markdown (§9). `MemorySpaceService.export_space` gathers the corpus once (index + every entry's bytes) behind the viewer+ permission gate, including the member grant list only for editor+ callers (mirrors `list_members`). The app-api route builds the archive in a `SpooledTemporaryFile` — spilling to disk beyond 8 MiB so a large space never pins memory — and streams it back. Zip mirrors the S3 layout (`{name}/MEMORY.md`, `entries//.md`, `metadata.json`) so it is self-contained and re-importable later. Archive path components are sanitized against zip-slip. Route tests cover layout, verbatim frontmatter, owner-vs-viewer member disclosure, 403/404/flag-off, and the hostile-slug case. Co-Authored-By: Claude Opus 4.8 * feat(memory): Memory Space sharing + optimistic manifest concurrency (A4) The sharing leg of Workstream A, plus the concurrency guarantee that makes multi-editor spaces safe. Sharing surface (app-api, over the existing service grant methods): - GET /memory/spaces/{id}/shares list grants (editor+) - POST /memory/spaces/{id}/shares grant viewer|editor (owner) - PATCH /memory/spaces/{id}/shares/{email} change a grant's role (owner) - DELETE /memory/spaces/{id}/shares/{email} revoke (owner, idempotent) New `MemorySpaceService.update_share` gives PATCH proper not-found semantics and preserves the grant's original createdAt (distinct from share's upsert). Optimistic manifest concurrency (the real design content): - `MemorySpaceRepository.put_index(expected_version=…)` does a conditional DynamoDB write on the manifest `version`, raising the repository-local `OptimisticLockError` on a mismatch. - `write_entry`/`delete_entry` route through a new `_mutate_index` helper: a bounded read-modify-conditional-write retry loop. Because an entry write touches a single slug, re-reading the fresh manifest and re-applying is safe; it converges on transient races and raises `MemorySpaceConcurrencyError` (→ 409) only on a sustained one. Behavior is unchanged for single-writer spaces. Tests: 6 route tests (share CRUD, member gains access, non-owner 403, viewer can't list, PATCH-unknown 404, owner-role 422) + 7 service tests (update_share role/origin/owner-gate + version-increments, stale-write rejected, retry converges, gives-up-after-max). 76 memory + import-boundary tests green. Co-Authored-By: Claude Opus 4.8 * feat(memory): Memory Spaces SPA panel — list/detail/create/share/export (A5) The user-facing "Memory" surface for the Memory Space primitive, under frontend/ai.client/src/app/memory-spaces/. Makes A2–A4 visible to users. - List page: owned + shared-in spaces as cards with role/template badges; per-card open, share (owner), download .zip, delete/leave. Empty, loading, error, and feature-unavailable states. - Detail page: view/edit the MEMORY.md index (editor+) and the entry list; entries open in a dialog to view (viewer) or edit/create (editor+); delete per entry. Header carries share/download/delete-or-leave. - Create-from-template dialog and a share dialog (add-by-email + per-row role + delta-on-save over the A4 /shares endpoints). Viewer access is read-only throughout; the share dialog fails soft for non-owners. - Signal facade (MemorySpaceService) + thin API service mirror the assistants/schedules pattern. The nav entry rides a live accessible$ probe: a 404 (MEMORY_SPACES_ENABLED off) hides it, matching showSchedules. - Routes memory-spaces + memory-spaces/:id (authGuard); redesign-tokens and @angular/cdk/dialog conventions throughout. Facade spec (7 tests) green; dev build + tsc clean; sidenav specs still pass. Co-Authored-By: Claude Opus 4.8 * test(memory): mock MemorySpaceService in sidenav spec (fix unhandled rejection) The A5 sidenav now injects MemorySpaceService and probes `loadSpaces()` in the auth effect. The sidenav spec mocked ScheduleService but not the new service, so the authenticated-probe test constructed the real MemorySpaceService, which fired a real XHR to /memory/spaces (status 0, no backend); `loadSpaces` then re-threw, surfacing as a Vitest "unhandled rejection" at suite level. Mock MemorySpaceService exactly as ScheduleService is mocked, and add matching coverage: probe fires once authenticated (not while unauthenticated) and the `showMemorySpaces` nav gate resolves null→false, false→false, true→true. Full suite: 1422 passed, 0 errors. Co-Authored-By: Claude Opus 4.8 * fix(memory): wire memory-spaces table/bucket names onto app-api app-api owns the Memory Spaces CRUD surface (`/memory/spaces/*`) but its container environment only set MEMORY_SPACES_ENABLED — never the table or bucket names the service reads (DYNAMODB_MEMORY_SPACES_TABLE_NAME / S3_MEMORY_SPACES_BUCKET_NAME). Without them the repository falls back to the default "memory-spaces" table name, which doesn't exist, so every read throws a boto3 ResourceNotFoundException that the centralized handler maps to a 502 "Upstream service error." (inference-api already sets the identical trio, but per the service-boundary rule it isn't the one serving these routes.) Thread `refs.memorySpacesTable`/`.memorySpacesBucket` through AppApiSsmParams and emit both names next to the flag in buildAppApiEnvironment. Names are always wired (read lazily); only MEMORY_SPACES_ENABLED gates route mounting, so flipping the switch on later needs no env change. New unit test guards the wiring; tsc + 431 infra jest tests green. Co-Authored-By: Claude Opus 4.8 * docs(cdk): capture the "wire resource name to every compute" rule Fold the lesson from the app-api env-wiring fix into the cdk-infrastructure skill so the next construct-author sees it while wiring, not after a 502. Adds a "Cross-Construct References" subsection: set a resource's name env var on every compute that reads it (one doesn't imply the other), the silent-502 failure mode (default-name fallback → ResourceNotFoundException → generic 502, invisible to synth/CI), and the env-map test guard + service-boundary caveat. Co-Authored-By: Claude Opus 4.8 * feat(memory): deterministic consolidation health pass (A6) The safe, non-LLM slice of Workstream A6. `MemorySpaceService.consolidate` (editor+) + `POST /memory/spaces/{id}/consolidate` → a `ConsolidationReport`. Auto-fixes only storage hygiene: orphaned content-addressed objects — keys under a space's prefix that no manifest entry or the index pointer references (leaks from crashed/raced writes) — are GC'd (new `MemorySpaceStore.list_keys` drives it). Everything that needs a judgment call is *reported, not mutated*: - duplicate content across slugs (same content hash) — which slug survives is semantic, so it's flagged, never auto-merged; - dead `[[slug]]` wikilinks in MEMORY.md — reported; opt-in `stripDeadLinks` unlinks them (they point nowhere) while preserving the surrounding prose; - over-cap entry counts (`MEMORY_SPACE_INDEX_CAP`, default 200) — flagged, never auto-evicted. This deliberately does not merge/evict/rewrite durable memory — that's deferred to the LLM consolidation pass (Workstream B era), which extends this exact `consolidate()` seam once agentic writes create real duplication/staleness to act on. On-demand only for now; scheduler/threshold auto-run and SPA surfacing are follow-ups. Tests: 8 service (healthy report, orphan GC + skip, dup-report-no-merge, dead-link report + strip-keeps-prose, over-cap flag, editor gate) + 4 route (report shape, no-body, viewer 403, flag-off 404) + 3 store (list_keys prefix scoping / empty / disabled). 104 memory + boundary tests green. Co-Authored-By: Claude Opus 4.8 * docs(agent): Agent Designer spec — unified primitive-binding surface Captures the strategy for the "Agent Designer" (Agent Harness Editor): a new authoring surface that composes an Agent from RBAC-governed primitives (instructions, model, KBs, tools, skills, Memory Spaces, + future), replacing the term/feature "Assistant." Locks the load-bearing decisions: own a primitive-agnostic Agent contract and federate AgentCore Registry later rather than build on it (adopt-with-boundary precedent); evolve the assistant store in place (no parallel table); a uniform bindings[] model with the model as a governed single-select; RBAC = compose the five existing per-primitive access checks (incl. ModelAccessService), not a new system; design-time filter + run-time re-resolution per invoker with block-on- missing v1; ship memory-consumption as a thin vertical slice before the full Designer. Phasing 0–5 + later AWS federation; supersedes the memory spec's "extend the Assistant" §B1 framing. Co-Authored-By: Claude Opus 4.8 * feat(agents): Agent contract + compat mapping in shared assistants models Phase 1 (PR-1) of the Agent Designer. Pure library, zero behavior change: legacy Assistants read unchanged and no caller passes the new fields yet. - AgentModelConfig (D3 governed single-select; field is model_settings/ alias modelConfig to dodge pydantic's reserved model_config — R3) - AgentBinding (open kind on read, KNOWN_BINDING_KINDS for request validation) - optional model_settings + bindings on Assistant (additive) - compat.effective_bindings/to_agent_view (D2): absent bindings synthesize a knowledge_base binding reffing the assistant id (KB's only stable identity, F4 deferred — R4); absent model maps to None, never fabricated (R1) - Decimal-safe serialization for modelConfig.params floats Co-Authored-By: Claude Opus 4.8 * feat(agents): persist bindings + modelConfig with design-time validation Phase 1 (PR-2). The Agent fields now round-trip through the rag-assistants store and are validated at write time by composing existing RBAC checks (D4). Legacy clients are unaffected: the SPA sends none of the new fields and the AssistantResponse surface is unchanged. - service.create_assistant/update_assistant thread bindings + model_settings; to_ddb_safe on write / from_ddb on read so modelConfig.params floats survive DynamoDB (Decimal); explicit [] replaces bindings, absent leaves them untouched - app_api/agents/services/binding_validation.py composes model access (ModelAccessService), memory resolve_permission (viewer+/editor+), the implicit-KB rejection, and inert shape-only checks for tool/skill (D4/D5) - assistants POST/PUT validate then pass through; validation raises 4xx outside the create handler's generic except so it isn't masked as 500 - tests: validation matrix (incl. inert no-RBAC guarantee), persistence round trip, legacy no-field read Co-Authored-By: Claude Opus 4.8 * feat(agents): /agents alias router behind AGENTS_API_ENABLED (dark) Phase 1 (PR-3). A governed Agent read/write surface over the evolved assistant store: same shared service functions and identity-based access gates as /assistants, but returning the Agent shape (compat.to_agent_view -> AgentResponse) so callers see modelConfig + bindings. Legacy ids valid unchanged. - feature_flags.agents_enabled(): AGENTS_API_ENABLED, default OFF (memory-spaces pattern) — surface 404s while off, ships incrementally, /assistants unaffected - app_api/agents/routes.py: require_agents_enabled 404-gate; draft/create/list/ get/update/delete + 4 shares endpoints, delegating to apis.shared.assistants service and reprojecting via to_agent_view; create/update run binding_validation - AgentResponse/AgentsListResponse/AgentSharesResponse (agentId == assistantId) - main.py mounts the router - test-chat + document sub-routes deliberately excluded (would force a 2nd architecture import-boundary exception); list is owner+shared (public/pagination parity deferred to the Phase-4 Designer) - tests: 404-gate, agentId/bindings projection, CRUD permission gating, shares Co-Authored-By: Claude Opus 4.8 * feat(agents): wire AGENTS_API_ENABLED through CDK + Phase 1 docs Phase 1 (PR-4). Completes Phase 1 deployability: the /agents surface can now be turned on per environment. No new AWS resources — the flag only gates whether the routes 404 (the assistant store it reads is always present). - config.ts: AgentsConfig { enabled }; CDK_AGENTS_API_ENABLED (default off, empty-string-safe) or an `agents.enabled` cdk.json context, mirroring memorySpaces exactly - app-api-environment.ts: AGENTS_API_ENABLED env on app-api - infra tests: default-off / opt-on assertion; mock-config default - docs: agent-designer.md Phase-1 status (+ the two refinements and the Oliver-dogfood-gated-on-Phase-3 note); CHANGELOG [Unreleased] The live Oliver dogfood (D6) is deliberately NOT included: it needs Phase 3 harness resolution + Memory Spaces deployed to the target env before a memory_space binding resolves at invocation. Tracked as the Phase 3 payoff. Co-Authored-By: Claude Opus 4.8 * feat(agents): thread AGENTS_API_ENABLED to the inference runtime (Phase 3 PR-0) Phase 3 harness resolution runs inside inference-api, so the runtime needs the same flag the app-api surface got in #593. Default off, mirrors the app-api wiring; without it the harness ignores Agent bindings entirely (today's behavior). Co-Authored-By: Claude Opus 4.8 * feat(agents): resolve Agent modelConfig at invocation, per invoker (Phase 3 PR-A) The Harness now re-resolves an Agent's governed modelConfig against the INVOKING user (D5) and applies it to model selection. Absent modelConfig ⇒ the model resolves exactly as today; gated on AGENTS_API_ENABLED (off in all envs still). - agent_binding_resolver.py: resolve_agent_invocation() checks the pinned model against AppRoleService.can_access_model for the invoker (R2 — same gate the harness uses elsewhere), returns a model_override or raises AgentBindingBlockedError. inference-api imports apis.shared only (boundary-safe) - routes.py /invocations: resolve after assistant load, before the KB search; on block, stream a conversational stream_error via stream_conversational_message (D5 block-with-message, no silent downgrade). Override wins at model resolution; agent params sit beneath request params, still flowing through admin bounds/locks - tests: allowed→override, denied→block (checked vs invoker), no-modelConfig→ empty plan (no RBAC call); 57 existing inference/chat tests unchanged Co-Authored-By: Claude Opus 4.8 * feat(agents): Memory-Space hydration helper for prompt injection (Phase 3 PR-B) Shared, sync helper that resolves a memory_space binding's alwaysLoad specs into injectable text fragments — the read side of Workstream B. - resolve_always_load(): MEMORY.md → index; latest:/ → most-recent matching manifest entry (defines that scheme, which had no resolver); bare slug → entry. Missing entries skipped (never fails a turn). Byte-budgeted with a truncation marker pointing at memory_read (MEMORY_INJECTION_MAX_BYTES, ~24KB) - render_memory_block(): delimited system-prompt block; empty for a fresh space - reads go through MemorySpaceService (re-checks viewer+ internally) — no leak - 11 unit tests against a fake service Co-Authored-By: Claude Opus 4.8 * feat(agents): inject bound Memory Space into the prompt, per invoker (Phase 3 PR-C) The Harness now resolves an Agent's memory_space binding against the invoking user and injects the space's alwaysLoad content (read-only) into the system prompt — the first half of the Workstream B / Oliver payoff. - agent_binding_resolver: _resolve_memory() checks the invoker's grant via MemorySpaceService.resolve_permission (D4); blocks (D5) when the flag is off, the space is gone, or a readwrite binding meets a below-editor invoker (no silent read-only downgrade). Returns ResolvedMemoryBinding (v1: first binding) - routes.py: after prompt assembly, hydrate via resolve_always_load (asyncio.to_ thread; MemorySpaceService re-checks viewer+) and append render_memory_block; best-effort — a memory-read hiccup never fails the turn - tests: memory grant matrix (none/flag-off/missing/read-viewer/readwrite-viewer- block/readwrite-editor/invoker-identity); 78 inference+compat tests green Co-Authored-By: Claude Opus 4.8 * fix(agents): rename app_api.agents package to avoid shadowing top-level agents run-app-api.sh launches app-api with `cd src/apis/app_api && python main.py`, putting that directory on sys.path[0]. The new apis/app_api/agents/ package (Agent Designer surface, #591/#592) then shadowed the top-level `agents` package, so `admin/quota/routes.py`'s `from agents.main_agent...` resolved into it and crashed startup with `ModuleNotFoundError: No module named 'agents.main_agent'`. Tests never caught it — pytest runs from backend/ where `agents` resolves correctly. Production is unaffected (the container runs `uvicorn apis.app_api.main:app` from WORKDIR /app, so sys.path[0] is /app). Rename the package apis/app_api/agents → apis/app_api/agent_designer (and its test dir) so its name can't collide with the top-level `agents` package. Pure rename: the /agents URL surface, router, and behavior are unchanged. Verified: `import agents.main_agent.quota.repository` and the full app-api module now load from src/apis/app_api; 34 agent/boundary tests pass. Co-Authored-By: Claude Opus 4.8 * feat(agents): memory_* tools scoped to an Agent's bound Memory Space (Phase 3) Completes the Workstream B write side: an Agent with a memory_space binding now gets memory_list / memory_read (always) and memory_write (readwrite bindings only) at invocation — Oliver can read AND write his space. - agents/builtin_tools/memory_spaces/: closure-scoped factories capturing the binding's space id + invoker identity, MemorySpaceService via asyncio.to_thread (artifact-tools pattern). Every call re-checks the grant inside the service (viewer+ read / editor+ write), so a revoked grant becomes an error tool-result mid-session, never a leak - routes.py _build_memory_tools(): appended to the extra_tools seam only when a memory binding resolved; write tool gated on access==readwrite. extra_tools agents are never cached → tools closed over user A can't be served to user B - not gated on enabled_tools: the governing capability is the Agent's binding, not the user's tool picker (same reasoning as artifact tools) - tests: tool success/permission-error/not-found matrix + seam counts (none=0, read=2, readwrite=3); 84 inference+tool tests green Co-Authored-By: Claude Opus 4.8 * chore(memory): default Memory Spaces ON with a kill switch Memory Spaces is a complete feature (CRUD + SPA panel + agent binding), so it should ship enabled for every deployer/forker — opt-out, not opt-in — matching the kbSync / scheduledRuns convention. The table + bucket are already provisioned unconditionally in PlatformStack, so this only flips the runtime MEMORY_SPACES_ENABLED env var; no new infra footprint. - config.ts: memorySpaces.enabled default true (empty/unset workflow var = on, only literal "false" disables); interface + block docs updated - platform.yml: forward CDK_MEMORY_SPACES_ENABLED (kill switch) and CDK_AGENTS_API_ENABLED (per-env enable for the still-default-off /agents surface, so dev can turn it on for dogfooding without a code change) - app-api-environment.ts: comment reflects default-on - config.test.ts: 5 tests locking default-on + kill-switch + context override Agent Designer (AGENTS_API_ENABLED) stays default OFF until the Phase-4 Designer UI ships — a headless /agents surface helps no forker. Co-Authored-By: Claude Opus 4.8 * feat(agent-designer): Phase 4 — Agent Designer UI + bindable catalog API Ships the Agent Designer authoring surface (Phase 4) and its Phase-2 precursor, the bindable-primitives catalog. The pickers can't exist without the catalog, so both land together. Backend (Phase 2 catalog): - GET /agents/bindable?kind=model|tool|skill|knowledge_base|memory_space returns an RBAC-filtered palette, composing the 5 existing per-primitive access services (D4); no new RBAC invented. Uniform BindableItem shape so every picker consumes one contract. Route declared before /{agent_id} so the literal path isn't captured. knowledge_base → empty (welded/synthesized); skill/memory_space → empty when their feature flag is off. Behind AGENTS_API_ENABLED. - Fix binding_validation._validate_model: it resolved models via get_managed_model() — a primary-key lookup on the internal UUID — but modelConfig.modelId is the Bedrock model_id that the runtime resolver, RBAC (permissions.models) and invocation all key on. A valid model would have been rejected 400 on save the moment the picker set one. Now matches by model_id, consistent with the whole chain. Frontend (Phase 4 UI): - New agents/ feature dir (separate from the Assistants editor): Agent + Binding + BindableItem TS contracts, a thin AgentApiService, and an AgentService signal facade with the accessible$ 404-probe idiom + a per-kind bindable cache. - Agents list page (model + binding-count badges) and an agent-form page: persona/emoji/tags/starters, a required single-select model picker (D3), tool/skill multi-select chips, and a memory-space picker with access (read/read+write — write disabled unless editor+ on the space, per D5) and an alwaysLoad MEMORY.md toggle. KB shown read-only. Sharing reuses the assistants share dialog (agentId == assistantId). - Routes agents / agents/new / agents/:id/edit, plus a sidenav "Agents" entry gated on the accessible$ probe. Tests: backend 1552 pass (9 new catalog + 4 new route + 3 updated model-validation); SPA build + tsc clean, ng test 7 AgentService + 11 sidenav specs pass. Co-Authored-By: Claude Opus 4.8 * test(sidenav): stub AgentService probe to fix unhandled HTTP rejection The sidenav constructor now probes agent accessibility (void agentService.loadAgents()), but sidenav.spec.ts didn't provide a mock AgentService, so the real service fired an unstubbed HTTP GET /agents that rejected with status 0 — vitest fails the run on unhandled errors even though all assertions passed. Provide a mock AgentService (accessible$ signal + no-op loadAgents) mirroring the schedule/memory stubs, plus parity tests for the probe + showAgents gate. Co-Authored-By: Claude Opus 4.8 * fix(agent-designer): align model write-check with the bindable catalog The catalog lists models via ModelAccessService.filter_accessible_models, but design-time write validation used can_access_model — and the two disagree. filter_accessible_models grants access whenever the model id is in the user's AppRole permissions.models; can_access_model only honors that membership when the model record ALSO carries a non-empty allowed_app_roles. So a model granted purely via the user's AppRole (empty allowed_app_roles) was listed by the picker but rejected on save with a 403 — and the runtime resolver (membership-based) would actually have allowed it. Validate the model with the same filter_accessible_models predicate the catalog uses, so 'if the palette offers it, the write accepts it' holds by construction. Adds a regression test for the empty-allowed_app_roles grant. Co-Authored-By: Claude Opus 4.8 * feat(sidenav): gate Memory Spaces + Agents to system-admin, add Preview badges Match the Scheduled Runs treatment for the two other preview surfaces: Memory Spaces and Agents now also require the system_admin AppRole (showX() && isAdmin()) in addition to their accessibility probe. Adds a small amber 'Preview' badge to all three nav entries (Agents, Memory Spaces, Scheduled Runs) so their preview status is visible. Co-Authored-By: Claude Opus 4.8 * feat(agent-designer): resolve tool bindings at invocation (replace + per-invoker RBAC) An Agent's `tool` bindings were stored by the Designer but inert at run time — the free-select tool picker fully drove the toolset regardless of what the Agent bound. This resolves them, mirroring the shipped `modelConfig` override: - Run-time (inference-api): `resolve_agent_invocation` now returns `plan.tools` (`ResolvedTools`). When an Agent binds tools they *replace* the request's `enabled_tools` for the turn; each bound tool is re-checked against the INVOKING user via `AppRoleService.can_access_tool` (the same AppRole gate the harness uses for model, R2) and a missing tool blocks the turn with a message (D5). No tool binding ⇒ `plan.tools is None` ⇒ the request drives the toolset exactly as today. Wired at the existing `extra_tools`/`get_agent` seam via `effective_enabled_tools` (also feeds the spreadsheet/artifact tool gates + attachment guidance/inventory). - Design-time (app-api): `tool` dropped from `_INERT_KINDS`; a bound tool must be in the author's palette (`ToolCatalogService.get_user_accessible_tools`, the same source the picker fetches — "if the palette offers it, the write accepts it", cf. the model check). The palette is resolved once per write. `skill` bindings stay inert here (their run-time fold interacts with agent_type/skill resolution — a follow-up slice). Tests: 6 resolver cases (override, dedupe, block-on-missing, per-invoker, none→passthrough) + 5 validation cases (accessible/inaccessible/empty-ref/fetch-once/lazy). Full backend suite green (4621 passed). Co-Authored-By: Claude Opus 4.8 * feat(topnav): surface active assistant in the top nav Move the assistant/agent indicator out of the chat-input footer and into the top nav, beside the session title, so an attached assistant is visible throughout the conversation. - Add a compact 'variant' to app-assistant-indicator: a subtle name-only pill (emoji + name) that opens the same actions menu (New session / Edit / Share) on click. The full card style is preserved behind variant="card". - Add a menuPlacement input so the actions dropdown opens downward in the top nav instead of clipping off-screen. - Thread the assistant/owner/loading state and action outputs from the chat container into app-topnav; render the pill (with a loading shimmer) to the right of the title. - Remove the now-orphaned footer indicator and loading skeletons from the full-page chat container (embedded preview footer left intact). - Assistant card: move conversation starters into a collapsible accordion (expanded by default) to keep the card compact. Co-Authored-By: Claude Opus 4.8 * feat(agent-designer): resolve skill bindings at invocation (replace + force skill-mode) Completes the tool/skill runtime-resolution gap (tools landed in #601). An Agent's `skill` bindings were stored by the Designer but inert at run time. This resolves them, mirroring the tool/model overrides: - Run-time (inference-api): `resolve_agent_invocation` now returns `plan.skills` (`ResolvedSkills`). When an Agent binds skills they *replace* the request's skills for the turn AND the route forces `agent_type="skill"` so the SkillAgent discloses exactly the bound set. Each bound skill is re-checked against the INVOKING user via `AppRoleService.can_access_skill`; a missing skill — or the Skills feature being disabled in this environment — blocks the turn with a message (D5). No skill binding ⇒ `plan.skills is None` ⇒ the request's agent_type/enabled_skills drive the turn as today. Wired by reassigning `effective_agent_type`/`effective_skill_ids` before the main-turn get_agent, so the values flow into the construction snapshot and a bound-skill agent resumes on the same skills_hash (resume-safe, same mechanism the tool slice relies on). - Design-time (app-api): `skill` dropped from inert (no inert kinds remain). A bound skill is flag-gated (`skills_enabled()`) and must be in the author's palette (`resolve_accessible_skill_ids`, the same source the picker fetches — cf. the tool check); the palette is resolved once per write and only when skills are enabled. Tests: +6 resolver (override, dedupe, flag-off block, block-on-missing, per-invoker, none →passthrough) + 6 validation (accessible/inaccessible/empty-ref/flag-off/fetch-once/lazy). Full backend suite green (4631 passed). Co-Authored-By: Claude Opus 4.8 * feat(agent-designer): reflect governed agent bindings in the chat-input (lock pickers) The backend governs an Agent's model/tool/skill bindings at invocation (#601, #602) — the agent's set wins regardless of what the client sends. The chat-input still showed the model/tool/skill pickers as free-select, which was dishonest (a change the backend ignores). This locks each picker to the active Agent's bindings, per primitive. - Session page (`session.page.ts`): inject AgentService/ToolService/SkillService; fetch the governed Agent alongside the assistant (agentId == assistantId) in `loadAssistant`; apply per-primitive locks from `modelConfig`/`bindings`, and release them when navigating to plain chat. Best-effort: the /agents surface may be disabled (404) or the assistant may be a legacy assistant with no bindings — every failure leaves the pickers free-select. - ModelService/ToolService/SkillService: add a small agent-lock API (`lockToAgent*` / `clearAgentLock` + `agentLocked`/`agentModelLocked`). While locked, `enabledToolIds`/ `enabledSkillIds` return the bound set (replace semantics, matching the backend), toggles no-op, and `isToolShownEnabled`/`isSkillShownEnabled` render the bound set honestly. - UI: model-dropdown shows a locked read-only chip ("set by this agent"); model-settings shows a "Set by agent" model row and a "This agent uses a fixed set of tools/skills" banner, with tool/skill/sub-tool toggles disabled + greyed while locked. This is UI honesty, not enforcement — the backend remains the authority. Per-primitive: an agent that binds a model but no tools locks only the model; the rest stay free-select. Tests: +5 tool-lock, +5 skill-lock, +4 model-lock service specs (ng test, 51 pass); `tsc` clean; production build (AOT template check) clean. Known limitations (documented for follow-up): a model race if the pinned model isn't in the user's loaded set yet (dropdown disables but may show the fallback name until models load); the skill-lock banner only shows in skills chat-mode. Co-Authored-By: Claude Opus 4.8 * fix(agent-designer): release chat-input picker locks on new conversation The agent-binding picker locks live in root singleton services (Model/Tool/Skill Service) that outlive the session component. Clicking "New chat" navigates to `/`, which recreates the session component with fresh assistant()/agent() signals (both null). The lock-release lived inside the `if (loadedAssistant || … || agent())` guard, which is false on that fresh component — so the stale locks from the previous agent conversation were never released, leaving the model + tools pickers stuck. Move `clearAgentBindingLocks()` out of the guard so it always runs when there is no assistant in the URL. Idempotent — a no-op when nothing is locked. Co-Authored-By: Claude Opus 4.8 * feat(agent-designer): show only the bound tools/skills when an agent locks the settings When an Agent dictates a fixed toolset/skillset, the settings panel listed every accessible tool/skill with the bound ones toggled on and the rest greyed off — a long, noisy list. Filter to show ONLY the bound (enabled) tools/skills so the panel reflects exactly what the agent uses. - ToolService.visibleTools / SkillService.visibleSkills: agent-locked → filter to the bound ids; otherwise the full accessible list. - model-settings template iterates the visible* lists. Tests: +1 tool-lock, +1 skill-lock spec (ng test green); tsc + AOT build clean. Co-Authored-By: Claude Opus 4.8 * chore(agent-designer): default AGENTS_API_ENABLED on with a kill switch The Agent Designer is complete (contract → surface → resolution → Designer UI → binding reflection), so flip the feature flag from opt-in to default-on, matching the house style for shipped features (scheduled_runs / memorySpaces). - backend `agents_enabled()`: empty-string-safe default-on — unset/empty ⇒ enabled, only the literal "false" disables (was `== "true"`, default off). - CDK `config.agents.enabled`: mirror the memorySpaces/scheduledRuns ternary (`!== 'false'` + context fallback `?? true`), so an unset/empty GitHub Actions var can't silently disable it. - Tests: add the Agents API default-on/empty/kill-switch/context suite to config.test.ts (mirrors Memory Spaces); rename the app-api-environment threading test (no longer "default off"). The `/agents/*` API now ships everywhere; the SPA nav stays preview-gated (system-admin + "Preview" badge) until Assistants are deprecated, so this doesn't broaden user-facing exposure — it just stops the API 404ing per-environment. Co-Authored-By: Claude Opus 4.8 * feat(agent-designer): manage an agent's knowledge base from the Agent Designer Extract the assistant editor's inline "Knowledge base" section into a standalone, reusable KnowledgeBaseSectionComponent and use it in both the assistant form and the Agent Designer — replacing the agent form's read-only "managed automatically" card with the live document/web-crawl/connector flow. This closes the last Agent migration blocker. The gap was frontend-only: the document upload/ingestion/retrieval pipeline already keys on the record id and agentId == assistantId, so /assistants/{id}/documents backs an agent unchanged. No backend or data-model changes (Option 1, not the deferred F4 first-class KB primitive). The component owns record identity via a createDraft callback so the first content-adding action can mint a draft in create mode; a permissionResolved input gates the edit-only sync-policy calls so a viewer never 403s on the default owner guess. The assistant form keeps createDraftAssistant as its callback (shedding ~1000 lines); the agent form adds createDraftAgent and drops the read-only kbBinding path. Verified: ng build clean, ng test 1449 specs green (incl. assistant-form spec). Co-Authored-By: Claude Opus 4.8 * test(scheduled-runs): freeze dispatcher clock to de-flake cadence rearm test test_next_run_at_uses_schedule_cadence asserted the daily-9am re-arm delta fell in (1h, 48h), which fails when CI runs in the hour before 9am Boise (the next daily run is legitimately <1h away). Freeze dispatcher._now to a fixed instant and assert next_run_at equals compute_next_run_at recomputed from the same instant, making the test time-of-day independent. Co-Authored-By: Claude Opus 4.8 * feat(agent-designer): govern model params + live editor preview Model-params governance: - binding_validation._validate_model_params rejects params that are unsupported / locked / out-of-[min,max] / out-of-allowed against the model's admin supported_params (belt-and-suspenders to the runtime merge; author-facing 400 instead of a silent clamp). +9 tests. - Data-driven Parameters subsection under the model picker reading meta.supportedParams (numeric inputs, enum selects, locked read-only); empty params omit `params` (today's exact resolution). Live side-by-side preview in the agent editor: - New AgentPreviewComponent reuses PreviewChatService and streams the SAVED agent through the real /chat/stream invocation path, so all bindings (model/params/tools/skills/memory) resolve server-side. Capability strip + dirty banner make the resolved context and the save-to-apply semantics explicit. - Agents send a minimal request body (message/session_id/agent id) and opt out of the assistant preview's system_prompt + owner-tools injection, which fought the bindings and blew the 8KB system_prompt cap for long personas (422). PreviewChatService gains a backward- compatible opts flag; assistant preview behavior unchanged. - Two-column editor shell mirroring the assistant editor. Verified: backend 59 pass (9 new), ng build clean, 16 SPA specs (7 agents + 9 preview-chat). Co-Authored-By: Claude Opus 4.8 * feat(agent-designer): lock preview model picker; trim preview nav The Agent Designer preview reused the main chat-input, whose model dropdown reads the root ModelService — so it showed the user's global model (e.g. Sonnet 5) and let them switch it, even though the harness resolves the model from the agent's binding server-side. Wire the preview to lock that picker to the agent's model via the same lockToAgentModel mechanism the session page uses for a real agent conversation, released on destroy (and idempotently on the next plain chat via the session page's self-heal effect). Also hide the Memory Spaces and Scheduled Runs side-nav entries for now (routes/pages and their capability probes are unchanged, so re-enabling is just re-adding the template blocks). Agents stays system-admin only. Co-Authored-By: Claude Opus 4.8 * fix(memory-spaces): route namespaced entry slugs via :path converter Entry slugs are namespaced with a slash (e.g. `people/brian-bolt`), but the app-api entry routes declared a plain `{slug}` param whose converter stops at `/`. Uvicorn percent-decodes `%2F`→`/` before routing, so `/entries/people/ brian-bolt` never matched `/entries/{slug}` and returned 404 on view/edit/delete. Switch the GET/PUT/DELETE entry routes to the `{slug:path}` converter so the embedded slash is captured and the slug arrives matching the manifest. Adds a route test exercising upsert→read→delete with a slashed slug. Co-Authored-By: Claude Opus 4.8 * feat(memory-spaces): let the agent read/write MEMORY.md via reserved slug MEMORY.md is the space's human-readable index — a standalone S3 object outside the entries manifest, injected into the agent's context each session via hydration. It never appears in `memory_list`, and the agent had no tool to read it back or keep it in sync with the entries it writes, so the machine-readable manifest and the human-readable index could silently drift. Route the reserved `"MEMORY.md"` slug (case-insensitive) through the existing service methods: `memory_read("MEMORY.md")` → `read_index` (viewer+), `memory_write("MEMORY.md", body)` → `update_index` (editor+, body only). No new tool surface; matches the literal hydration already uses. The slug is reserved — the agent cannot create an ordinary entry named MEMORY.md. Write stays gated identically to entry writes (only bound when the binding grants readwrite; service re-checks editor+). Docstrings + spec §4/§5 updated. Co-Authored-By: Claude Opus 4.8 * feat(schedules): target Agents instead of Assistants on scheduled runs The scheduled-run form's target selector now lists Agents (the Agent Designer primitive that supersedes the Assistant) instead of Assistants. Same underlying record — agentId == assistantId — so the wire field stays `assistantId` and no backend change is needed for the swap. Because an Agent's `tool` bindings replace the run's `enabled_tools` at invocation (agent_binding_resolver / routes.py effective_enabled_tools), the manual tool picker is now hidden whenever an Agent is selected — showing it would let the user pick tools that get silently discarded. The picker (and its snapshot semantics) remains only for the "Default agent" case. Submit drops any stale snapshot when an Agent is targeted. Also fixes "Run now" to target the selected Agent via ragAssistantId (the /runs/now backend already accepts it) — previously it ignored the target, so the attended test surface didn't match what the schedule would actually run. Co-Authored-By: Claude Opus 4.8 * chore(kaizen): weekly research scan 2026-07-10 Generated by the kaizen-research skill. Top 5 ideas appended to docs/kaizen/review-queue.md for the kaizen-review-prep run later this morning. Co-Authored-By: Claude Opus 4.8 (1M context) * chore(deps): upgrade Strands to 1.47.0 and add aws-bedrock-token-generator Bumps strands-agents 1.40.0 -> 1.47.0 (and the [bidi] extra to match) and adds aws-bedrock-token-generator==1.1.0 (bounded >=1.1.0,<2.0.0 by strands' openai extra). strands-agents-tools stays at 0.5.2 (resolver-confirmed compatible). Unblocks Bedrock Mantle work that needs the newer SDK: - OpenAIResponsesModel (Responses API) for models that don't support Chat Completions (e.g. openai.gpt-5.x on Mantle). - bedrock_mantle_config, which mints the Mantle bearer token via aws-bedrock-token-generator and derives the base URL + model-family base path (openai.gpt-5.* -> /openai/v1, else -> /v1). Full backend suite green on 1.47.0 (2306 passed). Co-Authored-By: Claude Opus 4.8 * fix(scheduled-runs): remove RBAC gate causing prod 403 "Access Denied" Regular users hit a 403 "You do not have access to scheduled runs" toast on page load. The `/schedules` and `/runs/*` surfaces were gated by the `scheduled-runs` RBAC capability, granted only to a beta cohort's AppRole (admins passed via the `*` wildcard). The sidenav ran a background `loadSchedules()` probe on every load, and the global errorInterceptor popped the toast on the 403 before the schedule service's graceful catch ran. The feature doesn't need admin/beta gating — keep it low-key and reachable only by direct URL for now: - Drop the capability check from both `require_scheduled_runs_user` gates; only the `SCHEDULED_RUNS_ENABLED` kill switch remains (404 when off). Runs still execute with the caller's own RBAC-allowed tools, so this widens who can reach the surface, not what any one caller can do. - Remove the vestigial sidenav schedules probe and dead showSchedules/navigateToSchedules wiring (the template never rendered a "Scheduled runs" link). - Update route + sidenav tests accordingly. `apis/shared/rbac/capabilities.py` is now unreferenced; left in place as generic RBAC infra so re-gating is a two-line revert. Co-Authored-By: Claude Opus 4.8 * docs: consolidate release workflow into one auto-invoked steering doc + skill Fold the versioning and release-notes guidance into a single 'cutting a release' guide covering the branch workflow, SemVer bump + version sync, change identification across the divergent main/develop histories, writing both release docs, the squash-merge PR into main, and the required backmerge into develop. - Add .kiro/steering/cutting-a-release.md (inclusion: auto — name + description, intent-triggered) - Add .claude/skills/cutting-a-release/ (SKILL.md auto-invoked via description) with references/{release-notes-format,changelog-format}.md for progressive disclosure - Remove superseded .kiro/steering/{versioning,release-notes}.md and .claude/skills/{versioning,release-notes}/ - Repoint .github/copilot-instructions.md at the consolidated skill/steering * feat(models): Mantle Responses API + per-model region; drop endpoint-path knob Refactors the admin "mantle" provider onto Strands' bedrock_mantle_config so the SDK owns the base URL, model-family base path, and bearer-token minting — removing hand-rolled inference plumbing. Adds the two things the library can't infer as declarative per-model fields: - apiMode (chat | responses): selects OpenAIModel vs OpenAIResponsesModel. Some Mantle models (e.g. openai.gpt-5.x) only serve the Responses API and reject Chat Completions, which the endpoint-path knob could never satisfy. - region: optional override into bedrock_mantle_config["region"], driving both the Mantle endpoint host and the SigV4 region the token is signed for — so a model can pin inference to its host region (e.g. gpt-5.x in us-east-1) independent of where the app runs. mantleEndpointPath is kept as an accept-but-ignore deprecated schema field (no stored record breaks) and removed from the UI + runtime. The Responses API uses different native param names, so to_mantle_config selects a Responses map (max_output_tokens, nested reasoning.effort) by mode. Runtime fields (mantle_api_mode/mantle_region) thread through model_config, the agent factory, base_agent, the paused-turn snapshot, stream_coordinator, and the chat service/routes. get_mantle_base_url/generate_bedrock_bearer_token are retained for the admin model-browse list (not inference). Gemma 4 (google.gemma-4-31b) is temporarily un-curated: it needs the /openai/v1 base path but the SDK only routes openai.gpt-5.* there, and bedrock_mantle_config forbids a base_url override. Re-add once the "google.gemma-" family prefix lands upstream in strands-agents/sdk-python. Backend suite green (2342). Frontend typecheck + manage-models specs green. Co-Authored-By: Claude Opus 4.8 * fix(api-converse): serve /chat/api-converse from app-api, not via inference proxy The API-key converse endpoint was broken in cloud. app-api proxied POST /chat/api-converse to `{INFERENCE_API_URL}/chat/api-converse`, but inference-api now runs inside an AgentCore Runtime whose data plane only serves POST /invocations and GET /ping — any other path returns UnknownOperationException (404) before reaching the container. It worked locally only because localhost:8001 bypasses the runtime gateway. Relocate the handler onto app-api as a self-contained route (validate key -> RBAC -> bedrock-runtime.converse -> cost accounting), reusing the shared services it already depends on. app-api reaches Bedrock directly via its task role, so there is no inference-api hop and no INFERENCE_API_URL dependency. Delete the proxy, the now-dead inference-api route, and its DTOs (moved to app_api/chat/models.py). Repoint the converse tests at the app-api module. Verified: 263 backend tests pass, import-boundary test clean, and a real un-mocked smoke against a Bedrock model returns 200 (stream + non-stream). Co-Authored-By: Claude Opus 4.8 * fix(app-api): grant Bedrock streaming + inference-profile invoke The relocated /chat/api-converse handler calls Bedrock Converse from app-api, so the task role's invoke grant must cover what the catalog's model IDs need. Expand the BedrockInvokeModel statement to add bedrock:InvokeModelWithResponseStream (the stream=true path) and broaden resources to all-region foundation models plus the account-level inference-profile ARN, since the catalog uses `us.*` cross-region inference profiles. Mirrors inference-api's BedrockModelInvocation grant. Verified: infra tsc clean, 442 infra jest tests pass. Co-Authored-By: Claude Opus 4.8 * fix(settings): point API-key snippets at /api/chat/api-converse After the BFF refactor, CloudFront only routes /api/* to the backend; other paths hit the SPA origin, which rejects POST with a CloudFront 403. The generated curl/Python/JS examples emitted the bare origin, producing `/chat/api-converse`. Resolve a relative/empty appApiUrl against the current origin so snippets target `/api/chat/api-converse`; leave an already-absolute value (local dev's http://localhost:8000) untouched. Co-Authored-By: Claude Opus 4.8 * feat(api-converse): route Bedrock Mantle models via a shared builder The API-key /chat/api-converse handler was Bedrock-only; provider="mantle" models (e.g. openai.gpt-5.4) 400'd because it always called bedrock-runtime.converse. Add a Mantle path so the full model catalog works. Extract the Mantle model construction (class-pick + bedrock_mantle_config) and its param maps + MantleApiMode enum out of agents/main_agent/core into a new apis/shared/models/mantle.py, so the agent factory and the API-key handler share ONE implementation (app-api can't import agents/). The factory now delegates to build_mantle_model. The handler resolves the requested model's provider from the catalog and branches: bedrock -> boto3 converse (unchanged); mantle -> the shared builder + the bare Strands model's .stream(), which yields the same Converse-shaped events the Bedrock path already emits — so SSE translation and usage/cost accounting are shared (cost is now tagged with the real provider). Unknown / lookup-failure ids fail safe to the Bedrock path. Verified: shared builder + factory-delegation + handler mantle-path unit tests; and real dev-ai smokes — chat-mode Mantle and Responses-API Mantle (openai.gpt-5.4) both return 200 (stream + non-stream) against the live endpoint. Co-Authored-By: Claude Opus 4.8 * feat(app-api): grant bedrock-mantle:CreateInference for api-converse The api-converse Mantle path invokes a Mantle model directly from app-api, so the task role needs bedrock-mantle:CreateInference (Mantle's own IAM namespace) — without it, mantle requests AccessDeny. Fold it into the existing project-scoped Mantle statement (was browse-only Get*/List*), renamed BedrockMantleInference to mirror the runtime role's grant. Verified: infra tsc clean, integration jest green (24 passed). Co-Authored-By: Claude Opus 4.8 * feat(identity): MCP user identity forwarding via access-token enrichment Add an opt-in Cognito Pre-Token-Generation v2 Lambda that copies configured user-pool attributes into namespaced claims on the ACCESS token, so personalized MCP tools can identify the caller. The access token is the only token forwarded end-to-end to MCP servers, so enrichment needs no changes to the SPA -> app-api -> inference-api -> MCP forwarding path. Shipped disabled by default (opt-in): a fork that configures nothing gets zero resources and the token is forwarded as before. Enabling requires the Cognito Essentials feature plan (pinned on the pool) plus two GitHub Actions variables (CDK_MCP_TOKEN_ENRICHMENT_ENABLED + CDK_MCP_TOKEN_ENRICHMENT_CLAIMS), keeping the committed cdk.context.json inert. - config: McpIdentityConfig (enabled + accessTokenClaims); claim map settable via JSON env var or context; parseJsonRecordEnv helper. - handler: stdlib-only, fail-open Pre-Token-Gen v2 trigger (returns event unchanged on any error so login is never blocked). - construct: real-code Lambda (fromAsset) attached via addTrigger V2_0; pool featurePlan pinned to ESSENTIALS. - wired conditionally into PlatformStack; platform.yml job-level env. - docs: spec updated (open questions resolved) + implementation summary, incl. the mcp-servers follow-on handoff. Ref: docs/specs/MCP_USER_IDENTITY_FORWARDING_SPEC.md * fix(app-api): grant bedrock-agentcore:CreateTokenVault for OAuth provider create Admin "add OAuth provider" (POST /admin/oauth-providers/) returned a 502 Bad Gateway. dev-ai app-api logs showed the real cause: an AccessDeniedException on bedrock-agentcore:CreateTokenVault against token-vault/default. AgentCore's CreateOauth2CredentialProvider ensures the default token vault exists on the first provider create, which requires CreateTokenVault (+ GetTokenVault) on the caller. The app-api task role had the ...Oauth2CredentialProvider actions but not the TokenVault ones. The shared error handler maps an uncaught AWS ClientError to HTTP 502, so the missing permission surfaced as a 502 rather than a 403. Add CreateTokenVault + GetTokenVault to the AgentCoreWorkloadIdentityAccess statement. The resource scope (token-vault/*) already covered token-vault/default; only the actions were missing. Requires a platform.yml (CDK) redeploy to take effect. Co-Authored-By: Claude Opus 4.8 * fix(scripts): make sync-version.sh portable across GNU and BSD tools The version-sync script only ran inside the dev container / CI (GNU coreutils); on macOS (BSD sed/grep) it errored out and silently left the manifests un-synced, so a release cut locally had to hand-edit every manifest. Replace the three GNU-only constructs with POSIX equivalents: - `grep -oP ... \K` (Perl regex) -> `sed -n 's/.../\1/p'` / awk field split - `sed -i "expr"` (GNU in-place) -> sed_inplace helper (temp file + mv) - `sed "0,/re/s/..."` (GNU-only address) -> awk first-match replace Behavior is unchanged on GNU; the script now runs identically on macOS. Verified both --check and the write path (incl. shields.io `--` hyphen doubling and SemVer->PEP 440 lock conversion) round-trip on BSD tools. Co-Authored-By: Claude Opus 4.8 * feat(admin): make admin sidebar nav sticky on desktop Pin the admin layout aside below the sticky top bar so the section nav stays in view while the content area scrolls. Uses lg:self-start so the aside shrinks to its content (flex items stretch to full height by default, which defeats position:sticky), plus a max-height + overflow so a long nav scrolls internally. Mobile dropdown is untouched. Co-Authored-By: Claude Opus 4.8 * feat(frontend): redesign 404 page to match auth screens Rework the not-found page onto the same design system as the login and first-boot pages: the primary-derived lava-lamp parallax backdrop (six depth-tiered morphing blobs), the masked graph-paper grid overlay, and the frosted-glass card. The oversized 404 sits above the card where the auth pages place the logo, so all three screens read as one system. Preserves existing behavior (sidenav hide/show, Return Home, Go Back) and respects prefers-reduced-motion. Classes are nf-prefixed and component-scoped via view encapsulation. Co-Authored-By: Claude Opus 4.8 * fix(frontend): make shell scroll container real so sticky nav engages The admin aside's lg:sticky never engaged because its nearest scrolling ancestor was the app shell's `flex-1 overflow-y-auto` div, which had no bounded height — it grew to content and the window scrolled instead, so sticky bound to a box that never moved. Pin
to h-dvh so that div becomes a genuine scroll container; the admin aside and top bar now stick. Also apply the admin bar's frosted-glass treatment (bg-*/opacity + backdrop-blur-sm) to the session topnav so the two surfaces match. Co-Authored-By: Claude Opus 4.8 * docs(specs): quota cooldown windows + platform ceiling spec and committee one-pager Replaces the hard monthly quota cutoff with a three-layer model: anchored 5-hour cooldown windows (Claude-style, exact reset times), a hard admin-adjustable platform-wide monthly ceiling as the fiscal guarantee, and the per-user monthly limit demoted to a generous anti-runaway backstop with degrade-to-economy-model as the target behavior. Backstop horizon (monthly vs weekly) is a per-tier choice. Includes an admin pilot tuning playbook with an observe-only phase, a user-facing quota status endpoint, recommended opening numbers, and a 7-PR implementation breakdown. The one-pager is the committee-facing rationale. Co-Authored-By: Claude Fable 5 * fix(frontend): guarantee JIT compiler in vitest runs to stop PlatformLocation flake The unit-test builder keeps Angular packages external, so vitest evaluates raw fesm2022 chunks whose partial declarations (ɵɵngDeclareInjectable/ ɵɵngDeclareFactory) compile eagerly and require @angular/compiler. Its presence was incidental — loaded transitively via @angular/core/testing in the builder's init-testbed setup — so specs with no static Angular imports (app.spec.ts dynamic-imports './app') could evaluate an unlinked @angular/common chunk first and fail with "The injectable 'PlatformLocation' needs to be compiled using the JIT compiler, but '@angular/compiler' is not available" (angular/angular-cli#31993). - add src/test-setup.ts importing @angular/compiler, wired via the test target's setupFiles and included in tsconfig.spec.json - add src/test-setup.spec.ts guarding the invariant deterministically - bump the first shared-view.page spec to 15s: it pays the one-time dynamic page-chunk import, which can exceed 5s under full-suite load Co-Authored-By: Claude Fable 5 * fix(frontend): size chat scroll space to the response, adapt to shell scroll container Replace the fixed viewport-tall bottom spacer in the message list with a min-height on the last turn group (user message + its assistant responses). The response streams into the reserved space instead of pushing a static spacer further down: a short response leaves exactly the room needed to pin the user message at the top, and a response taller than the viewport leaves zero dead scroll below it. Turn groups are keyed by their first message id so a finished turn's DOM (including live MCP App iframes) never remounts when the next turn starts, and the end-of-conversation sections (loader, consent/approval prompts, compaction, orphan artifacts) render inside the reserved space so they stay visible next to the response. Also adapt the session page to the real shell scroll container introduced by #634 (frosted sticky nav): the window no longer scrolls, which had silently broken submit scroll-to-message and scroll save/restore. scrollToMessage now uses scrollIntoView with a scroll-mt-20 header offset, and save/restore reads the shell container's scrollTop via a stable #app-scroll-container hook. Co-Authored-By: Claude Fable 5 * feat(settings): make user settings sidebar nav sticky on desktop Mirror the admin layout change (#632): pin the settings aside below the sticky top bar so the section nav stays in view while the content area scrolls. Uses lg:self-start so the aside shrinks to its content (grid items stretch to full row height by default, which defeats position:sticky), plus a max-height + overflow so a long nav scrolls internally. Mobile dropdown is untouched. Co-Authored-By: Claude Opus 4.8 * feat(admin-tools): discover OAuth-gated MCP servers with the admin's vaulted token The admin tool "Discover" flow refused OAuth-gated MCP servers outright, so servers like the GitHub remote MCP server (api.githubcopilot.com/mcp/) could not be discovered — discovery either 400'd on auth_type=oauth2 or connected unauthenticated and got a 401 from the server (wrapped to a 400). Discovery now accepts the OAuth provider id and connects using the admin's own vaulted 3LO token for that provider, fetched via AgentCore Identity (get_token_for_user) and injected as a bearer — mirroring how the agent loop attaches the end-user's provider token at runtime, and reusing the exact path connector_status already uses. This validates the admin's own connection and lists the tools their token can see (providers such as GitHub scope-filter the tool list to the token's grants). It fetches the admin's token only; it cannot mint an arbitrary end-user's token. Backend: - Add requires_oauth_provider (alias requiresOauthProvider) to MCPDiscoverRequest. - Handler loads the provider, fetches the admin's vaulted token, injects it as oauth_token into create_external_mcp_client. requires_consent -> 409, unknown provider / conflict with forward_auth / oauth2-without-provider -> 400. Frontend: - Send requiresOauthProvider in the discover payload (the form control already existed) and the OAuth2CallbackUrl header (bare /oauth-complete, no query string) so the backend can resolve the admin's token. Tests: 5 backend tests for the OAuth-provider discovery path; 2 SPA specs for the discover payload. Co-Authored-By: Claude Opus 4.8 * feat(manage-models): add Sonnet 5 + GPT-5.4 curated cards, order by capability Add two curated model catalog cards: - Claude Sonnet 5 (bedrock, global.anthropic.claude-sonnet-5) — 1M context, effort-based reasoning, caching on. - GPT-5.4 (mantle, openai.gpt-5.4) — Responses API surface; the openai.gpt-5.* model id matches the SDK's /openai/v1 routing prefixes, so one-click create routes correctly (unlike the commented-out Gemma card). Order the Bedrock Claude cards most-capable-first (Opus 4.7, Sonnet 5, Sonnet 4.6, Haiku 4.5) and place GPT-5.4 ahead of Qwen in the Mantle list. Move the "Bedrock Mantle" provider tab next to "Bedrock" in the catalog selector. Co-Authored-By: Claude Opus 4.8 * fix(mantle): route google.gemma-4-* to /openai/v1 base path Gemma 4 is served ONLY on Mantle's /openai/v1 path (per its AWS model card), but the Strands SDK's _OPENAI_PATH_MODEL_PREFIXES ships only "openai.gpt-5.", so google.gemma-4-* fell through to /v1 and inference 401'd with access_denied ("... is not enabled for this account"). Append "google.gemma-4-" to the SDK's prefix table at build time (_ensure_gemma4_openai_v1_routing: lazy, idempotent, guarded) until it lands upstream. Scoped to the 4.x family — Gemma 3 stays on /v1. - Guard tests: prefix registers on build, all three Gemma 4 variants resolve to /openai/v1, Gemma 3 stays on /v1, registration idempotent. - Correct the stale curated-models.ts note (the "would fail at chat time" claim is obsolete; its "google.gemma-" re-add hint would have misrouted Gemma 3). - Add design note proposing mantleEndpointPath as a live admin setting as the durable alternative to chasing the SDK's hardcoded table. Co-Authored-By: Claude Opus 4.8 * feat(manage-models): make max output tokens optional Newer reasoning / Responses-API models (GPT-5.x, Claude with adaptive thinking) don't publish a discrete max-output-tokens value — output shares the context budget with reasoning tokens, so there's no fixed cap to enter. Our own GPT-5.4 curated card already carries a decorative value with no backing max_tokens spec. maxOutputTokens is only a ceiling for the admin-configured max_tokens inference param and is never sent to the provider, so leaving it unset is safe at inference time. This makes the admin form field optional to match. - ManagedModelCreate / ManagedModel: max_output_tokens -> Optional[int] - DynamoDB write: omit maxOutputTokens when absent (matches other optionals) - Form control: drop Validators.required, default null (number | null); the 0-default + min(1) combo would otherwise still block submit - SPA interfaces typed number | null; catalog card null-guarded (shows "— out") - Both ceiling validators already skipped an absent value — no change needed Co-Authored-By: Claude Opus 4.8 * fix(docker): float curl security patch to survive Debian mirror purges Debian removes the superseded point version of curl from the trixie mirror on each security update, so an exact +deb13uN pin breaks every build once the next CVE lands. Pin to +deb13u* to track the live patch while keeping the minor version fixed; the digest-pinned base image is what actually provides reproducibility. Co-Authored-By: Claude Opus 4.8 * feat(web-sources): allow removing a web source A web source could be added but never removed. There was no DELETE route, no client method, and no UI affordance — the only way to drop one was to delete every page document it produced and let the orphan cascade in cleanup_service pick up the crawl row as a side effect. Add the operation as a first-class one, inverting that existing cascade: DELETE /assistants/{id}/web-sources/crawls/{crawl_id} removes the crawl's sync policy, soft-deletes every page under its root URL (vectors and S3 teardown hand off to the same background cleanup the single-document path uses), then hard-deletes the crawl row. A crawl that is genuinely in flight is refused with a 409 rather than raced — the crawler would keep writing pages we just enumerated. A crawl stuck at 'running' because its process died is not in flight and stays deletable, so a zombie source can't become permanently undeletable. The route is edit-gated (owner or editor), matching the documents surface that renders the list. Co-Authored-By: Claude Opus 4.8 * fix(web-sources): let editors start and view crawls start_crawl, list_crawls and get_crawl gated on the owner-keyed get_assistant(), which returns None for a user holding only an editor share — so an editor got a 404 from the "Add web content" button the SPA already renders for them (canManageSync() shows it to anyone who isn't a viewer). Route them through the same _require_edit_permission helper the documents, sync-policies and delete-crawl surfaces use, so owner|editor is the gate and a viewer gets a 403 instead of a misleading 404. No owner_id threading is needed on these three: the document writes are keyed on the assistant (PK=AST#), not its owner, and imported_by_user_id/started_by_user_id intentionally record the *acting* user — substituting the owner there would credit an editor's import to the owner. owner_id stays confined to the delete path, whose soft_delete_document/_list_crawl_pages calls really are owner-keyed. Co-Authored-By: Claude Opus 4.8 * fix(rbac): make AppRole the single source of truth for model access The model admin page and the role admin page wrote to two different, unlinked fields. Enabling a model for a role on the model page wrote `allowedAppRoles` onto the model record — a field no access check ever read — so the grant silently did nothing: the role page still showed the model unchecked, and users never saw it in the chat picker. Only editing the role's `grantedModels` had any effect. Make the role record the single source of truth, matching the pattern tools and skills already use: - The model form's role picker now writes THROUGH to each selected role's `grantedModels` (new ModelRoleService.set_roles_for_model), mirroring set_roles_for_tool. Create/update/delete routes wire it up, migrating grants on a modelId rename and revoking them on delete. - `allowedAppRoles` is no longer persisted on the model; it is derived from the role records on read (hydrate_model_roles), so the model page and role page can no longer disagree. Adds `inheritedAppRoles` for wildcard/inherited grants, surfaced read-only in the form. - can_access_model and filter_accessible_models both delegate to one `_grants_access` predicate. They previously diverged (one gated on allowed_app_roles, one didn't), so a model could be listed by the catalog yet denied on use. - Removes the dead POST /sync-roles endpoint (never called; only existed to paper over the drift); replaces it with GET /managed-models/{id}/roles. - Drops Validators.required on the picker, since a model reachable only via a wildcard grant legitimately has zero direct grants. Adds regression coverage for the write-through, the derived read, and the two access checks agreeing. Full backend + frontend suites green (the 8 pre-existing get_metadata_storage failures are unrelated). Co-Authored-By: Claude Opus 4.8 * fix(tests): repoint storage patch target and harden integration gate Two pre-existing failures on develop, both unrelated to the code under test: - test_cache_savings.py patched apis.app_api.storage.get_metadata_storage, but that accessor moved to apis.shared.storage (the app_api.storage module is now an empty stub). Repoint all 5 patch targets. Production code in sessions/services/metadata.py already imports from the new location. - test_compaction_integration.py gated its real-AWS integration tests on AGENTCORE_MEMORY_ID. That variable leaks into the process mid-suite when other tests reload apis.app_api.main (load_dotenv(override=True) injects a local backend/src/.env), so the tests ran order-dependently against invalid credentials instead of skipping. Gate on an explicit RUN_AGENTCORE_INTEGRATION_TESTS=1 opt-in instead. Full suite: 4771 passed, 6 skipped. Co-Authored-By: Claude Opus 4.8 * fix(chat): keep SSE stream open across tab switches @microsoft/fetch-event-source defaults to openWhenHidden:false, which aborts the SSE connection on visibilitychange-to-hidden and reopens it — issuing a fresh POST /invocations for the SAME turn — when the tab becomes visible again. That reopen happens inside the library, reusing the request and bypassing the SPA's per-session double-submit and streamId supersession guards. Because a client abort does not propagate through the AgentCore Runtime data plane, the original backend agent keeps running while the reopened one runs the same turn concurrently. Both persist tool-use/tool-result events to the same AgentCore Memory session, corrupting history with duplicate / interleaved toolResult turns and bricking the conversation with a Bedrock "toolResult blocks exceed toolUse blocks" ValidationException. Set openWhenHidden:true on both fetchEventSource call sites so a single stream stays alive across tab switches (also correct for long agentic turns). The server-side restore-time repair is the safety net for already -corrupted histories. Co-Authored-By: Claude Opus 4.8 * fix(sessions): repair tool-use/tool-result pairing on restore Bedrock Converse rejects any history where a user turn's toolResult blocks do not exactly match the preceding assistant turn's toolUse blocks ("The number of toolResult blocks at messages.N exceeds the number of toolUse blocks of previous turn"). A single such violation anywhere in a session's persisted history makes every subsequent turn fail, permanently bricking the conversation. Such corruption can be written by concurrent/interrupted turns with parallel tool calls (e.g. a duplicate invocation spawned by a tab switch): duplicate toolResult turns, toolResults reordered away from their toolUse turn (assistant/assistant/user/user), or toolResults orphaned after a synthetic error turn. The SDK's own _fix_broken_tool_use only rebuilds the single message after each toolUse turn, so it does not repair these shapes. Add TurnBasedSessionManager._repair_tool_pairing, an unconditional restore-time normalizer (sibling to _strip_document_bytes) that rebuilds a Bedrock-valid history on the final agent.messages: every toolUse turn is immediately followed by exactly one matching result turn (missing ones synthesized as errors), duplicate/orphaned result turns are dropped, and consecutive same-role turns are merged. No-op (identity) on healthy history. Kill switch: AGENTCORE_MEMORY_HISTORY_REPAIR_ENABLED=false. Validated against a real bricked production history (24 violations -> 0, idempotent). Self-heals affected sessions on their next turn. Co-Authored-By: Claude Opus 4.8 * test(sessions): make compaction fixtures valid Converse histories Two pre-existing compaction tests fed the session manager a user toolResult turn with no preceding assistant toolUse (make_tool_result_message alone) — an invalid Converse history that Bedrock would also reject. The new restore-time _repair_tool_pairing correctly drops/merges those orphaned turns, changing the message counts the tests asserted. Give each fixture a matching toolUse turn before the toolResult so the repair no-ops and the tests exercise compaction/truncation and checkpoint slicing in isolation. Counts updated accordingly (4->5 kept; slice 2->3). Co-Authored-By: Claude Opus 4.8 * fix(sessions): guard synthetic error persistence against role-alternation breaks When a turn errors inside the agent stream, the handler persists a synthetic "⚠️ Something went wrong" assistant turn. If the last persisted message was already an assistant turn (a dangling assistant toolUse, or a prior synthetic error turn), this appends a second consecutive assistant message, breaking Bedrock's strict user/assistant alternation. The next turn then fails and persists yet another assistant error turn — an amplifier that turns one bad turn into a permanently bricked session. Add a centralized role-alternation guard in persist_synthetic_messages via a new last_persisted_role param: any synthetic turn that would land adjacent to a same-role turn is dropped (the error stays a live-only UI affordance, the same choice the max_tokens path already makes). Callers in stream_coordinator pass the history tail role via a new _last_persisted_role(agent) helper. Complements PR #653's restore-time _repair_tool_pairing, which masks this on the model-request path; this fixes the write side so storage and the message display stay clean too. Co-Authored-By: Claude Opus 4.8 * fix(chat): reject duplicate concurrent turns with a per-session single-flight lease A client-side abort (Stop, tab switch, dropped socket, retry) does not propagate through the AgentCore Runtime data plane, and the Runtime can route a duplicate POST /invocations to a different container. Two agent loops then run concurrently against one AgentCore Memory session and corrupt tool-pairing history, which Bedrock Converse rejects on every subsequent turn ("toolResult blocks exceed toolUse blocks"). This bricked prod session f761f59b. Follow-up to PR #653, which closed the frontend tab-switch vector. Add a distributed single-flight guard at the inference-api /invocations turn-start chokepoint: - session_lease.py: acquire/renew/release on a dedicated sessions-metadata item (PK=USER#{uid}, SK=LEASE#{sid}) via an atomic conditional write. leaseExpiresAt is the app-level check; ttl is a coarse auto-reap backstop. Owner-scoped renew and release. Fail-open on any non-conflict DynamoDB error. - routes.py: acquire at turn-start; reject a duplicate with 409. Resume / max-tokens continuation re-enter an already-ended loop, so they acquire with force=True (never blocked, still install a lease). Heartbeat renews the lease while the turn streams; release in the generator finally + both except handlers. Preview / no-DynamoDB paths skip the guard. - SPA: handle the 409 as a soft "Already responding" notice (AlreadyStreamingError) instead of a hard "Chat Request Failed" toast; unwrap the BFF's double-encoded detail; loading clears so the user can retry once the prior turn finishes. Design note + distributed-cancel follow-on: docs/specs/session-single-flight-guard.md Co-Authored-By: Claude Opus 4.8 * feat(chat): distributed turn cancellation — make Stop actually stop the server turn Follow-on to the single-flight lease (#655). A client abort doesn't propagate through the AgentCore Runtime data plane, so Stop was cosmetic server-side: the container ran to completion, held the lease, and burned model/tool spend. That left "Stop → resend" returning 409 until the prior turn finished naturally. Reuse the lease as the cross-container signalling channel: - Signal: the app-api user_stopped endpoint calls request_session_cancel, which stamps cancelRequestedFor= on the lease item (owner-scoped, so a stale Stop can't kill a later turn). Best-effort — never fails the Stop. - Observe: the inference-api lease heartbeat (tightened 30s→10s) renews with ReturnValues=ALL_NEW and, on cancelRequestedFor==owner, flips session_manager.cancelled. - Effect A (tools): the always-on StopHook cancels the next tool call. - Effect B (model stream): a cooperative check at the top of the StreamCoordinator loop raises _CooperativeStopSignal; a dedicated arm persists the partial via _persist_interruption (marked user_stopped), emits terminal SSE frames, and ends cleanly (no re-raise) so a still-connected client closes and the lease releases. This is what ends a pure-chat turn, which has no tool boundary for StopHook. - acquire clears any stale cancel marker (REMOVE) on takeover. Net: Stop ends the server turn; the 409-on-resend window shrinks from a full turn to ~one heartbeat (10s), and wasted spend after Stop is halted. Rides the existing interrupted-turn teardown/persist path (hardened by #653's _repair_tool_pairing), so stopping mid-stream never orphans or corrupts history. Residual (documented): in-flight tool calls finish before cancel is seen; already- generated Bedrock tokens are billed. Design note: docs/specs/session-single-flight-guard.md Co-Authored-By: Claude Opus 4.8 * fix(infra): grant app-api task role access to shared-conversations table The shared-conversations DynamoDB table was threaded into the app-api container as an env var (SHARED_CONVERSATIONS_TABLE_NAME) but never granted on the task role. Every conversation-share operation therefore failed against DynamoDB: - POST /conversations/{id}/share -> PutItem AccessDeniedException - GET /conversations/{id}/shares -> Query AccessDeniedException on the SessionShareIndex GSI Both surfaced to users as a generic 500 "Failed to create share". Add `SharedConversationsAccess` to the app-api coreTables grant list so the role gets the standard DynamoDB action set on the table and its GSIs (index/*), matching every other table the app-api touches. Also add a regression test that synthesizes PlatformStack and asserts the app-api role has a SharedConversationsAccess statement granting PutItem/Query/GetItem with a GSI resource and no wildcard. Verified the test fails without the grant. Co-Authored-By: Claude Opus 4.8 * feat(shares): offload large-conversation snapshots to S3 Sharing a large conversation failed: ShareService.create_share inlined the full message list into a single DynamoDB item, exceeding the 400 KB item limit and surfacing to users as a bare 500 (observed in prod-ai as a PutItem ValidationException). This is separate from the IAM-grant bug in PR #657. Offload the snapshot body (messages + metadata) to a new private shared-conversations S3 bucket, keeping only control fields plus a body_ref pointer in DynamoDB — mirroring the Memory Spaces / Artifacts / Skills S3-offload pattern. Reads fall back to inline for legacy shares, so existing shares keep working with no migration and the SPA contract is unchanged. - New ShareSnapshotStore (content-addressed S3 put/get/delete, SSE-S3, dedupe) - create_share writes body to S3 + body_ref item; revoke/session-cleanup best-effort delete the object - _load_snapshot_body reads from S3 or falls back to legacy inline items - ShareStorageUnavailableError -> friendly 503 instead of a bare 500 - CDK: shared-conversations bucket + SSM param, compute-ref, app-api env (SHARED_CONVERSATIONS_BUCKET_NAME), and SharedConversationsBucketReadWrite IAM grant (app-api only) - Tests: store round-trip/dedupe, >400 KB regression, S3 + legacy reads, export-from-S3, revoke cleanup, storage-unavailable Spec: docs/specs/share-large-conversations-s3-offload.md Co-Authored-By: Claude Opus 4.8 * fix(agents): resolve model provider for agent-bound invocations Agent (assistant) model bindings persist only `model_id` — never `provider` — so previewing/invoking an agent bound to a Mantle model (e.g. `openai.gpt-5.4`) resolved to provider=None. That misroutes the model to Bedrock ConverseStream, which rejects it with "The provided model identifier is invalid", even though the same model works from the normal chat path (which always sends `provider` alongside `model_id`). Two complementary fixes: - Backend (server-authoritative): `_resolve_model_settings` now also returns the model's registered `provider` from the managed-model registry, and the invocation path backfills `effective_provider` from it when the request/binding didn't carry one. This fixes all existing agents with a provider-less stored binding — no data backfill needed — and mirrors how `mantle_api_mode`/`mantle_region` are already recovered. The app-tool-call / app-context-update rebuild paths get the same fallback so a rebuilt agent keys on the same provider as its main turn. - Frontend: the Agent Designer save payload now persists the selected model's `provider` (from the catalog `meta.provider`) alongside `modelId`, so newly created/edited bindings are self-describing. Co-Authored-By: Claude Opus 4.8 * fix(inference): bind effective_enabled_tools on resume path Resume turns (interrupt_responses set — OAuth-gated MCP consent or tool-approval) crashed with `NameError: cannot access free variable 'effective_enabled_tools'`. The variable is referenced unconditionally by the `stream_with_quota_warning` streaming closure (attachment guidance + tabular inventory) but was only assigned in the non-resume branch. On resume the closure raised before its first yield, the inference-api container returned 500, and the AgentCore Runtime data plane translated that into a 424 Failed Dependency to app-api and the SPA. This broke every interrupt-resume turn since the agent-designer tool-binding refactor (0b9b039a) — most visibly "connect to Gmail for employees", which completes via an OAuth-consent resume. Bind effective_enabled_tools from the paused-turn snapshot on the resume branch (the same source the resume get_agent call uses). Adds a resume-path regression test to tests/routes/test_inference.py that drives /invocations with interrupt_responses and asserts a 200 stream; without the fix it fails with the NameError. Co-Authored-By: Claude Opus 4.8 * chore(kaizen): weekly research scan 2026-07-17 Generated by the kaizen-research skill. Top 5 ideas appended to docs/kaizen/review-queue.md for the kaizen-review-prep run later this morning. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: add session-metadata static sort key spec (issue #175) Root-cause spec for the SessionMetadata parse-failure warnings: the session row's sort key encodes lastMessageAt, forcing a delete+put row move every turn. Concurrent writers race that move and upsert bare ghost rows. Also drives the first-turn duplicate-row race. Fix: static SK (S#{session_id}) + sparse SessionRecencyIndex GSI for recency listing. Covers the expand -> migrate -> backfill -> contract migration, downstream/forked-deployment safety (marker gate + graceful GSI-missing fallback), pagination-token compatibility, and the test matrix. Co-Authored-By: Claude Opus 4.8 * feat(infra): add SessionRecencyIndex GSI to sessions-metadata (issue #175 Phase 0) Sparse recency index (GSI4_PK=USER#{id}, GSI4_SK={lastMessageAt}#{session_id}, projection ALL) for newest-first active-session listing once the base sort key becomes static. Phase 0 of the static-sort-key migration: adding the index is a no-op until rows populate GSI4 keys, so it deploys safely ahead of any code change. IAM already covers it via the SessionsMetadataAccess /index/* wildcard. Update tables-detailed test to assert all four GSIs (the "2 GSIs" title was already stale after DueScheduleIndex) and the new index's key schema. Co-Authored-By: Claude Opus 4.8 * feat(sessions): dual-scheme union read for session listing (issue #175 Phase 1a) Expand-read step of the static-sort-key migration. list_user_sessions now reads the UNION of two disjoint sources so a session is visible whether or not its base sort key has been migrated to the static S#{session_id} form: - legacy (un-migrated): base table, SK begins_with 'S#ACTIVE#' - migrated: SessionRecencyIndex GSI (GSI4_PK=USER#{id}, GSI4_SK={lastMessageAt}#{id}) Pagination switches to a value cursor ({lastMessageAt}#{session_id}) so each page is derived independently from the last returned position, with no cross-page buffering; fetching limit+1 valid rows per source is provably enough to detect a next page. The cursor decoder is tolerant — legacy/undecodable tokens fall back to first page (a harmless reset across the deploy boundary). Degrades to legacy-only if SessionRecencyIndex doesn't exist yet (code ahead of the CDK GSI): the GSI query's ResourceNotFoundException is caught. No writes change and no row migrates in this phase — this only teaches every reader to cope with both schemes, which must be fully rolled out before Phase 1b turns on self-migrating writes. Tests: union ordering, cross-union pagination (no dupes/gaps), migrated-only via GSI, ghost/preview skip, and graceful fallback when the index is absent. conftest sessions_metadata_table fixture gains the SessionRecencyIndex GSI to match prod. Co-Authored-By: Claude Opus 4.8 * fix(deps): bump strands-agents to 1.48.0 for cachePoint-attachment fix Auto prompt caching (CacheConfig strategy=auto) appended its cachePoint after the last user message's content, so any turn attaching a non-PDF document (txt/docx/csv/...) sent [text, document, cachePoint] and Bedrock's Anthropic adapter rejected it with "ValidationException ... messages.N.content.M.type: Field required", surfacing to users as "Agent force-stopped" (prod incidents Jul 14-16, e.g. session dd1a647a on a .txt transcript upload). strands 1.48.0 places the cache point before the first non-PDF document block instead (upstream issue #1966); every placement it produces was verified live against global.anthropic.claude-sonnet-4-6 ConverseStream. Also corrects the model_config comment that credited PR #1438/1.39.0 with this fix - #1438 was the auto-caching feature itself. Co-Authored-By: Claude Fable 5 * fix(sessions): degrade to legacy-only on real ValidationException for missing GSI (issue #175) The Phase 1a dual-scheme read (PR #667) catches a missing SessionRecencyIndex to fall back to legacy-only listing, but only handled ResourceNotFoundException — what moto raises. Real DynamoDB raises ValidationException ("The table does not have the specified index") for a missing GSI (verified against the prod table). So if the 1a backend deployed to an environment before the CDK GSI existed, list_user_sessions would 503 instead of degrading. Broaden the catch to also handle ValidationException (scoped by the "specified index" message so genuinely malformed queries still surface). This restores the intended order-independence: 1a is safe whether or not SessionRecencyIndex exists yet, which matters for prod deploy ordering (backend.yml vs platform.yml) and for forked deployments. Add a test that reproduces the real ValidationException on the index query (moto masks it), asserting fallback to legacy-only results. Co-Authored-By: Claude Opus 4.8 * Add Word document tools (create/modify/list/read) Provision a full Word (.docx) toolset behind the single create_word_document capability toggle. Each tool runs python-docx in Bedrock Code Interpreter and uses the existing user-files store (S3 + DynamoDB) for persistence and delivery. - create/modify/list/read tools in agents/builtin_tools/word_document_tool.py, injected per-request via _build_word_document_tools (inference_api/chat/routes.py). - Frontend inline-visual 'word_document' renderer with an accessible download button (Tailwind utilities, no scoped CSS). - Restore-time content-block sanitizer in TurnBasedSessionManager: drops empty/typeless blocks from restored history that caused Bedrock ConverseStream 'messages.N.content.M.type: Field required'. - Seed create_word_document in bootstrap DEFAULT_TOOLS ('Word Documents') + updated seed tests. * feat(sessions): static-SK write path — born static, self-migrate, no rotation (issue #175 Phase 1b) Turns on the write side of the static-sort-key migration. Sessions stop encoding lastMessageAt in the sort key, so the row never moves and the ghost-row race that produced "Failed to parse session item" warnings is structurally eliminated for every migrated row. Changed (all resolve the row via GSI, which is SK-scheme-agnostic): - ensure_session_metadata_exists: new sessions born at static SK S#{id} + GSI4 keys, with a real attribute_not_exists(PK) conditional put. The deterministic SK makes the guard meaningful, closing the first-turn duplicate-row race the old timestamped SK made impossible to gate. - update_session_activity: drops the per-turn Phase-B rotation. Static rows update in place (SET GSI4_SK re-positions the sparse recency index — no row move); a still-legacy row does its one-time final rotation to the static SK, carrying any concurrent write. - _store_session_metadata_cloud: static SK; migrate legacy->static on move; SET GSI4 for active, REMOVE for deleted; never un-migrates a static row. - session_service.delete_session: resolves the raw SK via _get_session_by_gsi instead of reconstructing S#ACTIVE#{lastMessageAt}#{id} (which misses migrated rows). Non-rotating soft-delete: SET status=deleted + REMOVE GSI4 in place, or migrate a legacy row to a static tombstone. Drops the S#DELETED# prefix (nothing reads it). The ~10 other writers resolve-then-update-in-place on the current SK and need no change — they already work on a static SK and never rotate. Tests: TestWriteSideMigration (born-static, one-time migrate, no rotation, soft-delete in-place/legacy, end-to-end create->activity->list->delete) plus the real ConditionalCheckFailedException contract (moto raises it for a failed conditional put). Updated three tests that encoded the old rotation contract. Full shared+routes suites: 1689 passed. Co-Authored-By: Claude Opus 4.8 * Fix S3 PutObject PermanentRedirect in Word tools The user-files S3 client pinned its endpoint to https://s3.{AWS_REGION}.amazonaws.com. In the AgentCore Runtime AWS_REGION does not reliably match the bucket region, and the explicit endpoint_url disables botocore's automatic S3 region redirect, so PutObject failed with PermanentRedirect. Resolve the bucket's real region via HeadBucket (x-amz-bucket-region header; maps to s3:ListBucket, which the runtime role already has — GetBucketLocation is not granted) and pin the client to it, dropping the hardcoded endpoint_url. Fixes both the save and the presigned download URL region. * feat(scripts): static-SK backfill for the cold tail + ghost cleanup (issue #175 Phase 2) One-shot, idempotent, throttled backfill that finishes the migration for rows the lazy write-path (Phase 1b) hasn't touched: rewrites legacy S#ACTIVE#/S#DELETED# session rows to the static S#{id} scheme (populating GSI4 for active, none for deleted), deletes the ghost/stub rows the old rotating-SK writers produced, and — only once a fresh scan finds zero legacy rows — writes the migration-complete marker that unblocks Phase 3. Safety: - Dry-run by default; --apply required to write. - Static put uses attribute_not_exists(SK) so it never clobbers a row a live writer already migrated with fresher data; the legacy delete is an idempotent no-op if already gone. - --sleep throttles; re-runnable to convergence. - Marker gated: --set-marker re-scans and withholds the marker while any legacy row remains, so Phase 3 can't be unblocked on partially-migrated data. Tested: 10 moto cases (classification, dry-run no-op, active+deleted migrate, ghost delete, idempotency, conditional-put skip of a live-migrated row, marker gating). Also validated as a dry-run against real dev-ai data (145 legacy rows, 0 ghosts) to confirm the real-DynamoDB scan/filter behavior. Co-Authored-By: Claude Opus 4.8 * feat(sessions): contract session list to GSI-only once migration completes (issue #175 Phase 3) Final phase of the static-sort-key migration. list_user_sessions now reads the SessionRecencyIndex GSI alone once the Phase 2 backfill has set the migration- complete marker (PK=MIGRATION#session-sk, SK=STATE, complete=true) — the legacy S#ACTIVE# union branch is only queried until then. - The marker check is memoised per-process (the marker only ever goes unset->set, never back), so migrated deployments pay no extra read after the first observation; a container that started pre-backfill picks up the flip on a later call. - Fails open: any error reading the marker keeps dual-read. Downstream/forked deployments that haven't run the backfill stay in dual-read, so removing the legacy branch here can never blank an un-migrated sidebar. - Safety net: even with the marker set, if the GSI query itself errors (transient ValidationException/ResourceNotFound) the legacy branch is still queried, so a flaky index never returns an empty list. The legacy code path is retained behind the marker rather than deleted, per the downstream-safety design; a later release can drop it once all deployments report the marker set. Tests: dual-read when marker absent, GSI-only (legacy row excluded) when set, marker memoisation, and GSI-failure-falls-back-to-legacy-even-with-marker. Full sessions+routes+backfill+architecture suites: 136 passed. Co-Authored-By: Claude Opus 4.8 * docs(specs): skills v2 — skills as a pure knowledge primitive bound on Agents Supersedes the tool-binding sections of admin-skills-rbac-tool-binding.md and the mode-toggle design in skills-mode.md. Skills become agentskills.io knowledge bundles (no bound_tool_ids), bound on Agents via the Designer, with a user-uploaded tier, opt-in chat selection, Strands AgentSkills plugin runtime, and invoke-through sharing for shared Agents. Co-Authored-By: Claude Fable 5 * refactor(agents): delete MCP tool-folding machinery and hook shims (skills v2 PR-1) Skills no longer bind tools, so the entire fold stack that served bound_tool_ids goes: mcp_binding.py (FoldedMCPTool, resolve_mcp_bindings, the two folded-tool lookup factories), mcp_tool_folding.py and its drop_folded_tools calls in FilteredMCPClient / UICapableMCPClient, the tool_use_provider_lookup / tool_use_approval_lookup shims on OAuthConsentHook / MCPExternalApprovalHook (+ FoldedToolApproval), and their wiring in base_agent. SkillAgent is neutered, not deleted (PR-2): DB-backed skills are instructions-only; the file/dev @skill binding path is unchanged. SkillRegistry loses all_bound_tool_ids/bind_catalog_tools. Note: the spec's stream_coordinator skill_executor unwrap item has no corresponding code — nothing existed to delete there. Per docs/specs/skills-as-agent-primitive.md §2/§8 (PR-1). Co-Authored-By: Claude Opus 4.8 * refactor(skills): remove bound_tool_ids end-to-end (skills v2 PR-1) A skill is a pure knowledge bundle: drop bound_tool_ids from SkillDefinition, the create/update/response DTOs and the Dynamo (de)serialization; delete _validate_bound_tools and the boundToolCount projection; stop emitting boundToolIds in the bindable-catalog meta; seed web_research as an instructions+reference-file bundle with no bound tool. SPA: remove the boundToolIds field/model plumbing, the skill-form bound tools section, the list-page badge, and the tool-picker dialog. Existing rows with boundToolIds deserialize fine (attribute ignored); no production data binds tools to skills (SKILLS_ENABLED=false everywhere). Co-Authored-By: Claude Opus 4.8 * refactor(chat): remove skills mode — policy, toggle, and preference (skills v2 PR-1) Skills are opt-in per turn, not a mode. Delete the admin chat-mode policy (platform_settings sentinel + admin /settings/chat routes + public /system/chat-settings), _resolve_effective_agent_type, and the preferred_agent_mode user setting. DEFAULT_AGENT_TYPE flips to "chat"; an explicit agent_type="skill" (future picker / agent-binding resolver) still resolves skills, and the enabled_skills request plumbing + skills_hash caching are kept verbatim for the PR-4 picker. SPA: delete ChatModeService and the Skills/Tools capabilities toggle; Skills + Tools sections render unconditionally (Skills gated on having skills or an agent lock, so it stays inert while the feature is off); stop sending agent_type/enabled_skills; drop preferredAgentMode and the session-preference agentType plumbing. Agent-bound skill locks now load the skill list on demand so locked rows render their names. Co-Authored-By: Claude Opus 4.8 * docs(specs): mark skills-mode and tool-binding specs superseded by skills v2 Co-Authored-By: Claude Opus 4.8 * feat(skills): swap runtime to Strands AgentSkills plugin (skills v2 PR-2) Spike gate PASSED (20/20, docs/specs/skills-v2-pr2-spike-findings.md): the vended AgentSkills plugin composes with our prompt assembly (block-level injection preserves cache points), the skills_hash cache key, paused-turn resume, and agent.state round-trips our TurnBasedSessionManager unchanged. Runtime swap: - Map DB SkillDefinition -> strands.Skill via a new skills/strands_mapping.py (slugged name for agentskills.io validity + S3/harness portability; true skill_id + human display_name carried in metadata). Add advisory allowed_tools + skill_metadata frontmatter passthrough to the model (D1/D4). - ChatAgent conditionally adds AgentSkills(skills=[...]) when the turn carries accessible_skill_ids; AgentFactory.create_agent gains a plugins param. - Retire the homegrown disclosure stack: SkillAgent, skill_registry.py, skill_tools.py (skill_dispatcher/skill_executor), the @skill decorator + file/dev definitions, and their tests. "skill" stays a registered alias -> ChatAgent so the existing agent_type/skills_hash cache-key + resume path (which the spec keeps) resolve to a ChatAgent-with-plugin unchanged. read_skill_file (L3 reference bytes) and the S3 SKILL.md write-through projection follow as separate PR-2 commits. Full backend suite green (4611 passed, 3 skipped). Co-Authored-By: Claude Opus 4.8 * feat(skills): standard bundle layout + read_skill_file (skills v2 PR-2) Completes the PR-2 runtime: reference-file disclosure (L3) and the agentskills.io bundle layout that makes each skill a portable artifact. S3 bundle layout + SKILL.md projection (#6): - SkillResourceRef gains `kind` (reference|script|asset); dynamo round-trips it, old rows default to reference. - resource_store keys files at skills/{id}/{references|scripts|assets}/{filename} (path-based, dedupe dropped for the readable standard layout); add put_skill_md and resource_key/skill_md_key helpers. - create_skill/update_skill write a SKILL.md projection generated from the row (new apis/shared/skills/bundle.py: slugify + generate_skill_md) — best-effort, never fails the catalog write. Admin upload route gains an optional `kind`. read_skill_file (#4): - New per-turn tool (agents/main_agent/skills/strands_mapping.build_skills_runtime returns plugin + tool from one record fetch). Resolves `path` against the skill's manifest (no filesystem/traversal), serves bytes from SkillResourceStore, labels scripts inert (D5), describes binary assets instead of dumping bytes, and is implicitly access-gated (bound only to the turn's effective records — richer §6 invoke-through is PR-4). Skill.instructions gain an "Available reference files" listing so the model knows what to request. - ChatAgent wires read_skill_file alongside the AgentSkills plugin. slugify moved to apis/shared/skills/bundle (shared by the app-api projection and the agents runtime; import-boundary safe). Full backend suite green (4634 passed). Co-Authored-By: Claude Opus 4.8 * docs(specs): mark skills v2 PR-2 done in the plan Co-Authored-By: Claude Opus 4.8 * fix(agents): correct stale skills copy and surface invalid-save feedback Two issues found while smoke-testing skills v2 PR-2 (#681). Skills v2 decision D1 removed skill->tool binding entirely: a skill is a pure knowledge bundle and never grants or carries a tool. Two surfaces still claimed otherwise -- the Agent Designer skills section ("with their own bound tools") and the admin skill edit form ("and bound tools"). Saving an invalid agent form also no-opped silently: persist() marked the controls touched and returned, so the only feedback was an inline error that is usually below the fold once the author has scrolled to the Model/Skills sections. The dirty banner stayed up, making the click look ignored. Now an invalid save also toasts and scrolls the first invalid control into view. The reveal helper matches on input/textarea/select rather than [formControlName]: the starters array binds [formControlName]="$index", a property binding that renders no attribute to select on. Verified with npm run build and ng test (127 files, 1452 tests passing). Co-Authored-By: Claude Opus 4.8 * feat(skills): user-authored skills tier (skills v2 PR-3) Adds the owner-scoped half of the skill catalog: any user can author their own agentskills.io knowledge bundles and reach them at runtime, without an admin RBAC grant. Backend - `list_skills_by_owner` — GSI4 (SkillOwnerIndex) partition query, the "list my skills" path. `list_skills` gains an `owner_id` filter. - `UserSkillService` — owner-scoped CRUD. Ownership is resolved on every path; a skill you do not own is 404, never 403, so the surface never confirms someone else's skill exists. Resource handling (caps, manifest, bundle layout, orphan GC) delegates to SkillCatalogService so both tiers emit identical bundles. - `/skills/mine/*` routes on the existing session-auth router, so they inherit the SKILLS_ENABLED mount gate. - Skill ids are allocated server-side from the display name and suffixed on collision (docx -> docx_2). Ids stay globally unique because the runtime activation key is the slugified id, and a 409 would disclose the existence of a skill the user cannot see. - `resolve_accessible_skill_ids` now returns catalog ∪ own — ownership is its own grant. This is what makes an authored skill usable at all. Two tier boundaries closed, both of which would have leaked private skills: - `get_all_skill_ids` (RBAC "*" wildcard expansion) now lists only catalog skills; a wildcard grant must not sweep in other users' authored skills. - The admin role-grant endpoints refuse user-authored skills, since granting one to an AppRole would hand a private document to a whole role. The admin catalog list is likewise scoped to owner_id == "system". Frontend - My Skills page (list + create/edit form) with SKILL.md import prefill and reference/script/asset uploads; scripts are labeled non-executable. - Nav entry gated on the same 404 accessibility probe memory-spaces uses. Co-Authored-By: Claude Opus 4.8 * fix(skills): preserve SKILL.md frontmatter through import (skills v2 PR-3) Live clickthrough found an imported bundle losing everything outside name/description: a SKILL.md carrying `license: MIT` round-tripped back out of S3 without it. Spec D2 requires import/export to be round-trip-faithful, and the backend already had `skill_metadata` + `allowed_tools` columns for exactly this — only the client-side import never populated them, leaving both fields dead on the user tier. - `parseSkillMarkdown` now also returns `allowedTools` (comma-separated or inline-array forms) and `metadata` (every non-reserved frontmatter key). Additive to the DTO, so the admin form is unaffected. - The My Skills form carries both through create and update, including for a loaded skill, so an edit never silently drops them. - Advisory tools render as chips with the D1 disclaimer — skills never grant tools; the bound agent decides. Also fixes a `capitalize` on the staged-file line title-casing the whole string ("Uploads When You Save"); only the kind should capitalize. Verified: a bundle with license + compatibility + allowed-tools now emerges from the S3 SKILL.md projection intact. Co-Authored-By: Claude Opus 4.8 * chore(skills): backfill script for v1 skill bundles (skills v2 PR-3) Skills authored before PR-2 have neither the SKILL.md write-through projection nor the standard bundle layout — v1 stored resources content-addressed (skills/{id}/{sha256}) with no `kind`. Their S3 prefix is therefore not a valid agentskills.io bundle: it can't be handed to a managed Harness or exported as-is, which is the whole point of the projection. The script fixes both per skill: copies each legacy object to its standard path, rewrites the row's manifest to point there (adding `kind`), and writes the SKILL.md generated from the row. It imports `generate_skill_md` and the key helpers from the live write path, so a backfilled bundle is byte-identical to one the app writes today. Follows the backfill_session_static_sk conventions: dry-run by default, idempotent, throttled, scopeable to one skill. Copies are non-destructive — legacy objects survive unless --delete-legacy, and a manifest entry whose bytes are missing is left untouched rather than repointed at nothing. Applied to dev-ai/web_research: manifest now points at references/extraction_tips.md, SKILL.md written, read path verified at 200. Co-Authored-By: Claude Opus 4.8 * fix(skills): seed example skill as a v2 bundle (skills v2 PR-3) Backfilling dev's web_research row surfaced that the seeder itself still emits v1 shapes, so every fresh environment reproduces exactly the state the backfill just repaired: - resources landed at the content-addressed key (skills/{id}/{sha256}) with no `kind`, instead of skills/{id}/references/{filename} - no SKILL.md was ever written, so the seeded prefix was not a valid agentskills.io bundle and could not be handed to a managed Harness or exported as-is Both fixed. The slug and frontmatter rules are duplicated from apis/shared/skills/bundle.py rather than imported, because seed.sh runs this script standalone after infra deploy without the app package on the path — the existing content-hash logic was duplicated for the same reason. Tests now pin the standard layout and the projection, plus a guard that the seed prose never again names the retired v1 meta-tools (skill_executor / skill_dispatcher). That drift is what left dev's row instructing the model to call tools deleted in PR-1/PR-2. Note the seeders are skip-if-exists, so this repairs new environments only; existing ones need backfill_skill_bundles.py (dev-ai: applied). Co-Authored-By: Claude Opus 4.8 * feat(skills): selection surfaces + invoke-through access (skills v2 PR-4) Wires the chat opt-in picker end-to-end and lands the §6 invoke-through access predicate. Per spec docs/specs/skills-as-agent-primitive.md §8. The picker's markup shipped in PR-1 but nothing ever sent its selection, so the plain-chat skills path had never actually run. Turning it on surfaced three latent bugs: - Skill resolution was gated on agent_type == "skill", so the picker could never have taken effect. Skills are now driven by the selection on any turn; agent_type gates nothing ("skill" stays a ChatAgent alias only so stale SPA sessions don't 422). - The binding resolver gated on AppRoleService.can_access_skill, which has no ownership clause — an author was blocked on their own authored skill when invoking their own Agent — and whose "*" wildcard matched any id at all, including another user's private skill. Clauses 1+2 now route through resolve_accessible_skill_ids, which expands "*" over the catalog only. - Paused-turn resume and the construction snapshot both keyed skills off agent_type == "skill", which would have orphaned the paused agent of any plain-chat turn carrying skills. Both now key off the snapshot's own enabled_skills. Invoke-through (D7) is deliberately AGENT-scoped: it lives in the binding resolver, not in resolve_accessible_skill_ids, because widening the shared resolver would leak an Agent owner's private skills into every invoker's plain-chat picker and bindable palette. The owner-match clause blocks chain-sharing, and a system-owned Agent gets no invoke-through at all so RBAC stays the sole gate on catalog skills. D6 default flip: an absent or empty enabled_skills means no skills, on both the runtime filter and the picker's untouched-preference default. The two must agree or the UI would show skills as active that the turn never loads. It also keeps skills free for turns that don't want them — an absent selection short-circuits before any RBAC or skill-table read. read_skill_file needed no per-call predicate: its record set IS the turn's effective skill set, so there is no id the model can name to reach a skill the invoker cannot use. The Designer palette union needed no code — /agents/bindable already delegates to resolve_accessible_skill_ids, which PR-3 widened. Backend 4714 passed; SPA 1469 passed across 128 spec files. Co-Authored-By: Claude Opus 4.8 * chore(assets): add GitHub connector logo variants Light/dark Octocat marks alongside the existing google-* connector logos. Follows the repo's theme convention: -light is the black glyph (for light backgrounds), -dark the white one. Nothing references these yet — they're staged for a GitHub connector. Unrelated to the skills work in this branch; riding along rather than sitting untracked. Co-Authored-By: Claude Opus 4.8 * feat(skills): enable Skills v2 + admin-only capability gate (skills v2 PR-5) Flips SKILLS_ENABLED to default-ON with a kill switch and adds the infrastructure wiring it never had, closing out the Skills v2 epic (spec docs/specs/skills-as-agent-primitive.md §8). SKILLS_ENABLED had zero CDK/workflow plumbing, so "enable it per environment" was not previously expressible. Adds SkillsConfig to config.ts, threads SKILLS_ENABLED into both app-api and inference-api (they must stay in step — design-time refuses to bind a skill while the flag is off, so a mismatch would let an Agent be built with skills the runtime then blocks), and forwards CDK_SKILLS_ENABLED in platform.yml with the empty-string-safe ternary an unset GitHub variable requires. Feature existence and audience are two independent controls. The flag says the feature exists in an environment; the new `skills` RBAC capability says who sees the user-facing surfaces. system_admin holds it implicitly via its "*" tools grant, so the picker and My Skills stay admin-only during rollout; GA is one grant of `skills` to the `default` role, no redeploy. The gate raises 404, not 403. The SPA hides the My Skills nav entry by riding the list call, so a 404 hides the surface while a 403 surfaces an error toast — the failure mode that got the scheduled-runs capability gate reverted in prod. It also deliberately does not gate the runtime: an Agent shared to an ordinary user must still resolve its bound skills (invoke-through, §6/D7), and a capability check there would break exactly that path. Verified live end to end against a real agentskills.io bundle (Anthropic's docx) uploaded as a user skill and bound to an Agent: L1 8,075 -> L2 9,835 (skills tool, SKILL.md body) -> L3 10,737 (read_skill_file on references/LICENSE.txt). Invoke-through confirmed with a second non-admin account — the grant resolves through the shared Agent while the same skill stays absent from that user's own picker and /skills/mine. The "session auth, not Bearer" assertion now walks the transitive dependency tree rather than each route's direct dependencies, since the routes hang off the capability gate which in turn depends on the session. Pins the invariant that actually matters instead of the shape. Co-Authored-By: Claude Opus 4.8 * refactor(rbac): remove dead AppRoleService.can_access_skill Skills v2 moved skill authorization to apis/shared/skills/access.py (resolve_accessible_skill_ids = catalog ∪ own, and resolve_invocable_skill_ids which adds the Agent-owner invoke-through clause). AppRoleService.can_access_skill has had zero production callers since PR-4 and is wrong on two axes for anything user-tier: no ownership clause, and its "*" wildcard matches ANY skill id including another user's private authored skill. - Delete the method and its three tests. test_can_access_skill_with_wildcard asserted can_access_skill(user, "any_skill") is True — it enshrined the wildcard over-expansion bug as expected behavior. - Fix two stale docstring/comment references in agent_designer's binding_validation.py that still named it as the live run-time mechanism; since PR-4 that is resolve_invocable_skill_ids. - Reword the intentional "deliberately NOT can_access_skill" rationale in skills/access.py and agent_binding_resolver.py to past tense so they no longer imply the function still exists. Co-Authored-By: Claude Opus 4.8 * refactor(skills): delete dead SkillAccessService Skills v2 moved skill authorization to apis/shared/skills/access.py (resolve_accessible_skill_ids / resolve_invocable_skill_ids). SkillAccessService was left behind with zero live callers — not exported from admin/services/__init__.py, not DI-wired, and reached by no dynamic import. Its can_access_skill carried the same "*"-wildcard over-expansion flaw that got AppRoleService.can_access_skill deleted in #686. Also updates the stale comment in admin/skills/routes.py that named the service as the consumer of the all-skill-ids snapshot; that snapshot is still live, but its reader is now skills.access. Co-Authored-By: Claude Opus 4.8 * docs(agents): draft the Agent Directory spec A browse-and-discover surface for published Agents: a directory page and a detail page modeled on the ChatGPT/Claude plugin-detail layout, built as a read-view over the Agent record rather than a new primitive. The central decision (D1) is that we do NOT introduce a "Plugin" noun. Both vendors need a bundle layer because their capabilities install into a workspace separately from any persona; our Agent's `bindings[]` already IS that bundle, attached to the persona. Every field a vendor plugin-detail page renders already exists on our record. Grounding findings that shaped the design: - `VisibilityStatusIndex` (GSI2) is live and populated, and PUBLIC *access* still resolves to "viewer" — only the *listing* was switched off in ad4437e9 when email sharing superseded a public index. The read path is mostly built. - Listing is nonetheless a new sparse GSI5, not a re-enable: `VISIBILITY#PUBLIC` is one hot partition and can't be filtered by category. GSI5 is the next free slot (GSI4 is DueSyncIndex), and DueSyncIndex on the same table is the precedent — unlisted agents have no key, so the query physically can't see them. - `listed` is deliberately separate from `visibility` (D3). Deriving listing from PUBLIC would retroactively publish every existing PUBLIC agent to the whole institution with no author consent; backfill is listed=false. - Publishing amplifies Skills v2 invoke-through from a typed email list to everyone, so the publish dialog must enumerate which authored skills it exposes, and memory_space bindings block publication outright (D5). - The Designer's block-on-missing rule (D5) strains under open browsing, so the detail page previews per-invoker runnability up front — which resolves the "per-invoker capability preview" open question parked in agent-designer.md. Also notes a behavior change needing a call: GET /agents/{id} currently returns `instructions` to any PUBLIC viewer, which is a much larger exposure once agents are broadly listed than it was under link-sharing. Co-Authored-By: Claude Opus 4.8 * refactor(artifacts): collapse the two catalog rows into one Artifacts toggle Artifacts shipped as two independent `protocol: local` catalog rows — `create_artifact` and `update_artifact` — so the tool picker listed them as two unrelated entries. The picker only groups children under a parent for MCP protocols (driven by `serverTools`), so there was nothing to nest them under; the flat listing was a data-model fact, not a template gap. Adopt the Word-documents idiom already used one entry below in the seed list: a single catalog row whose id is the gate key, with the runtime injecting the full toolset. `create_artifact` is now that key and provisions both the create and update tools. `seed_default_tools` is create-only, so a seed run does nothing to an environment seeded before this change. Add a backfill script that retitles the surviving row, promotes `update_artifact` to `create_artifact` everywhere a grant can hide — role TOOL_GRANT# items *and* the grantedTools/effectivePermissions.tools arrays on DEFINITION, user toolPreferences, assistant bindings — then deletes the retired row. Promote-before-delete, so an aborted run degrades to "both granted", never "neither". Schedule snapshots are left alone: a stale id there is an inert no-op since the runtime only reads `create_artifact`. Two judgement calls worth recording: - User prefs are a sparse override map, so only an explicit *enable* of the retired id carries over. Someone who switched update off while leaving create at its default-on never asked to lose artifacts, so an explicit disable just drops the key. - A role granting `*` gains no concrete grant — the wildcard already covers the keeper and narrowing it would be a silent scope change. Behavior change: anyone with create enabled but update disabled now gains update. That is inherent to collapsing the toggle. Co-Authored-By: Claude Opus 4.8 * chore(sidenav): hide the My Skills nav entry The /my-skills route, page, service, and the whole /skills/mine backend surface stay fully functional — only the sidenav link is removed, until we decide how users should actually navigate to their skills. Drops the now-dead showMySkills computed, the MySkillService injection, and its loadSkills() accessibility probe (that call existed solely to decide whether to render the link). Co-Authored-By: Claude Opus 4.8 * docs(skills): draft the Skill Creator spec Adopts the authoring half of Anthropic's open-source skill-creator and scopes out the eval half, which assumes a filesystem, subagent spawning, and script execution — none of which exist here (scripts are inert by design). Evals hang off F1's headless lane instead. PR-1 ships the methodology as an admin-catalog skill with zero code, routing the handoff through the My Skills form's existing frontmatter parsing. PR-2..PR-4 add the missing primitive: agent tools that write the user's own skills, executing as the invoking user and re-checking ownership per call, mirroring memory_write. Depends on skill-bundle-import's nested paths and server-side SKILL.md parser rather than restating them. Co-Authored-By: Claude Opus 4.8 * feat(skills): drop the `skills` capability gate from user-facing routes The user-facing skills surfaces (`GET /skills/`, `PUT /skills/preferences`, and all of `/skills/mine/*`) hung off `require_skills_capability`, which 404'd anyone not holding the `skills` RBAC capability. Its job was to keep skills admin-only during the v2 rollout, with GA framed as "one grant of `skills` to the `default` role, no redeploy." That GA path does not work, for two independent reasons: 1. `default` is a *fallback* role. `resolve_user_permissions` consults it only when a user matches zero AppRoles (service.py "Step 3"); it is not merged alongside a matched role. Prod's `default` carries no JWT mappings at all, so granting there would reach only unmapped users — never the faculty/staff/student cohorts. 2. A capability id cannot be granted from the admin roles UI regardless. That form builds `grantedTools` from the tool catalog, and a capability is not a tool, so there is no way to select it and no free-text entry. Net effect: an admin who granted a catalog skill to a role would find it silently invisible to that role's users, with no in-product way to fix it. Remove the gate. Skills are governed by `SKILLS_ENABLED` per environment and by a role's `grantedSkills` per cohort — a complete model that the admin UI can actually operate. Both surfaces are already self-limiting: `GET /skills/` returns only what `resolve_accessible_skill_ids` grants (no grants means an empty list and no rendered picker), and every `/skills/mine/*` route is owner-scoped inside `UserSkillService`. The route-coverage control is kept rather than dropped, retargeted from "every route is capability-gated" to "every route requires a session" — the invariant that still matters now that the session dependency is the only thing between these routes and an anonymous caller. A second test pins the removal so the gate cannot creep back without also making capabilities grantable from the roles UI. Also corrects the "GA = grant to `default`" claim where it appeared in capabilities.py, infrastructure/lib/config.ts, and platform.yml, and notes that `SCHEDULED_RUNS_CAPABILITY` is itself unused (that gate was dropped after 403ing in prod), which leaves capabilities.py with no consumers. Co-Authored-By: Claude Opus 4.8 * refactor(rbac): delete the dead capabilities module `apis/shared/rbac/capabilities.py` has no consumers left. It defined two capability ids granted through the `grantedTools` axis: * `SCHEDULED_RUNS_CAPABILITY` was orphaned when the RBAC gate on `/schedules` and `/runs` was dropped after 403ing in prod. * `SKILLS_CAPABILITY` was the last live caller, removed in the preceding commit along with `require_skills_capability` and its 12 route deps. Delete it rather than keeping it as a reference. The mechanism it documented is not one we want reached for again: a capability id cannot be granted from the admin roles UI (that form builds `grantedTools` from the tool catalog and offers no free-text entry), so any gate built on it is operable only by hand-writing DynamoDB items. Both live gates were removed for that reason. Its docstring carried two findings that cost real effort to establish, so they move to `AppRoleService.resolve_user_permissions` — the code they actually describe — rather than dying with the file: 1. `default` is a *fallback*, not a universal role. Step 3 substitutes it only when a user matched zero AppRoles, and prod's `default` carries no `jwtRoleMappings`, so granting there reaches only unmapped users. The "GA = one grant to `default`, no redeploy" framing that appeared in several comments was wrong. 2. The roles-UI limitation above, recorded where someone would see it before routing a new grant through this axis. The algorithm list in that docstring is renumbered to match the code's own step comments, so the "Step 3" reference is unambiguous. Also corrects both `feature_flags.py` docstrings, which still described the now-deleted capability as the companion "who may use it" control: `skills_enabled` points at a role's `grantedSkills`; `scheduled_runs_enabled` notes the flag is now the only control and the routes are deliberately ungated. Co-Authored-By: Claude Opus 4.8 * fix(chat-input): make the textarea scrollable and reset it after submit Three defects in the chat input's auto-sizing: - The textarea carried `overflow-hidden`, so once content exceeded the visible area there was no way to scroll within it. - `onTextareaInput` set `height = scrollHeight` with no clamp. Past 200px the inline height kept growing while `max-height` capped the rendered height, leaving the two diverged and the scrollbar unreachable. The growth is now clamped in a shared `autoResize()`. - `submitChatRequest` cleared the value but left the stale inline height, so the input stayed expanded after sending. Adds `resetTextareaHeight()`. Also drops the `.chat-textarea` CSS block: it declared its own min-height/max-height/field-sizing but the class is applied nowhere in the template, so the rules were dead while contradicting the real inline styles. The JS path is now the sole sizing authority. The `isExpanded` signal goes too — it was written on submit and never read. Co-Authored-By: Claude Opus 4.8 * fix(compaction): make restored history byte-stable to preserve Bedrock prompt-cache hits Tool-content truncation previously ran on every session restore behind a sliding protected-turns window, so each new turn re-mutated the turn that just aged past the window. Bedrock prompt caching requires an exact prefix match, so this forced a full prefix cache re-write (~$2.5/MTok on a 35k-150k prefix) nearly every turn — costing far more than the read tokens truncation saved (evidence: prod session aecd387d, inter-turn prefix shrinkages of -382/-1035/-1513 with cacheRead=0 inside the cache TTL). Redesign: truncation is now driven only by a persisted truncation_anchor in the compaction state (sessions-metadata `compaction` attribute): - The anchor moves when the checkpoint advances (update_after_turn), where the slice already pays the single cache re-write — one mutation per compaction event instead of one per turn. - It also advances opportunistically at restore when more than cache_ttl_seconds (default 300s, AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS) have passed since the previous turn: the cache entry has expired anyway, so pending truncations are applied for free. - Legacy state records without the field default the anchor to the checkpoint, so retained history stops being mutated immediately on upgrade. - The compaction-failure path in initialize() now resets _compaction_state_loaded so update_after_turn re-loads persisted state instead of overwriting checkpoint/anchor with defaults. Removes the now-dead _find_protected_indices sliding-window helper and adds tests/agents/main_agent/session/test_compaction_stability.py asserting agent.messages is byte-identical across consecutive restores whenever no compaction-state change occurs. Co-Authored-By: Claude Fable 5 * fix(skills): deterministic skill ordering to preserve Bedrock prompt cache Skill records reached the AgentSkills system-prompt block in nondeterministic order, changing the prompt between turns of the same session and invalidating the Bedrock prompt cache (exact-prefix match) — forcing full cache re-writes on turns well inside the TTL. Two sources, fixed at three layers: - batch_get_skills returned raw DynamoDB batch_get_item response order; now sorted by skill_id. - resolve_user_permissions built grant unions as sets and returned list(set) — iteration order varies per process via hash randomization; tools/models/skills now sorted. - build_skills_runtime sorts records before constructing AgentSkills as defense in depth at the injection point. Regression tests force a descending batch_get_item response order and a reversed fetch order, both verified to fail without the fix. Co-Authored-By: Claude Fable 5 * fix(cache): add tools + system cachePoints so message-level misses read the stable prefix CacheConfig(strategy="auto") places exactly one message-level cachePoint; when its lookup misses, nothing is read and the whole prefix re-writes at the cache-write premium. One proven miss mode is structural: Anthropic's cache lookback checks only ~20 content blocks behind the breakpoint, so a wide parallel tool fan-out (18 parallel calls = ~38 new blocks) pushes the previous checkpoint out of range — prod session aecd387d observed cacheRead=0 / cacheWrite=134k mid-turn (~$0.34). Now the request carries 3 of Bedrock's max-4 cachePoints: - toolConfig tail via cache_tools="default" - system tail via SystemContentBlock list with trailing cachePoint (built in AgentFactory; the cache_prompt config key is deprecated) - last-user-message point via the existing auto strategy (which strips only message-level points, never system/tools ones) Both new points are gated on ModelConfig.bedrock_cache_points_supported() (mirrors Strands' _cache_strategy predicate) because unlike auto mode they would be sent verbatim to non-Anthropic models and rejected. Verified live on global.anthropic.claude-sonnet-5 via ConverseStream: simulated message-level miss reads the 6.5k-token system+tools prefix from cache (cacheRead=6497, cacheWrite=83) instead of re-writing it. CountTokens accepts the cachePoint-bearing request, so context attribution is unaffected. Position/budget test asserts exactly 3 cachePoints. Upstream: strands-agents/harness-sdk#3348 proposes auto mode keep a rolling pair of message cachePoints so fan-outs stay within the lookback. Co-Authored-By: Claude Fable 5 * feat(observability): make prompt-cache economics measurable per model call A $1.60 prod conversation audit (session aecd387d) showed 75% of spend was avoidable Bedrock prompt-cache re-writes, and diagnosing the causes took hours of manual forensics against raw DynamoDB rows. This makes the whole class measurable end to end: - PrefixFingerprintHook (BeforeModelCallEvent) hashes the three cacheable prefix components per model call — toolConfig (order-sensitive canonical JSON), effective system prompt (captured after AgentSkills injection), and message history excluding the newest message. The stream coordinator persists entry N on the turn's Nth assistant-message cost row, so a miss is diagnosable with a column diff instead of row forensics. - Write-time cacheStatus per cost row (first_write | hit | miss_ttl_expired | miss_avoidable | uncached) derived from the session's previous C# row (one GSI read), plus wastedUsd for avoidable misses priced at the cache-write premium over cache-read from the row's own pricingSnapshot. Turn rows now write sequentially (was parallel) so each call classifies against its true predecessor. - Session-row rollups next to totalCost: totalCacheReadTokens, totalCacheWriteTokens, avoidableMissCount, wastedUsd — cache-efficiency ratio for lists/admin without scanning cost rows. - Admin cost anatomy: GET /admin/costs/sessions/{sessionId}/calls (require_admin) returns chronological per-call rows with token splits, cost, cacheStatus, and fingerprints, plus session-level cache summary. - CloudWatch EMF per call (CacheReadTokens / CacheWriteTokens / AvoidableMiss / WastedUsd) via a raw-JSON stdout logger — dashboard and alarm on fleet cache-write share with no SDK calls or extra IAM. - CI determinism guard: builds the chat-agent surface twice from shuffled skill/role/tool record orders (RBAC merge -> skills runtime -> Agent -> AgentSkills injection) and asserts identical system-prompt and toolConfig fingerprints; verified to fail when the skill-ordering sort is removed. Complements the sorting + byte-stability fixes from feature/prompt-cache-stability (merged in). Co-Authored-By: Claude Fable 5 * feat(observability): kill switch + prompt-cache conventions in CLAUDE.md PROMPT_CACHE_OBSERVABILITY_ENABLED=false disables the fingerprint hook, per-call cacheStatus derivation (and its GSI read), and EMF emission — default ON per house convention; empty string stays enabled. Raw cacheRead/cacheWrite token rollups are unaffected (usage passthrough, not derived). CLAUDE.md now encodes the determinism + byte-stability contract and the fingerprint-based debugging recipe. Co-Authored-By: Claude Fable 5 * chore: PR #697 follow-up breadcrumbs + token-cost-effectiveness tenet - CLAUDE.md: add "token cost effectiveness is a design tenet" bullet under Key Conventions — prefix determinism / bounded per-turn payloads / verify via the #697 observability, balanced so quality wins on genuine conflict. - kaizen review-queue: queue the two deferred #697 follow-ups — track harness-sdk#3348 (rolling-pair cachePoints; local workaround gated on dashboard evidence) and the ContextOffloader S3 adoption spike (with its four known gotchas). Co-Authored-By: Claude Fable 5 * feat(observability): prompt-cache CloudWatch dashboard + alarms (PR #697 follow-up) New cross-service construct area lib/constructs/observability/ with a PromptCacheObservabilityConstruct composed into PlatformStack. Graphs the dimension-less EMF metrics both APIs emit into AgentCoreStack/PromptCache (cache read/write tokens, a cache-efficiency MathExpression, AvoidableMiss, WastedUsd) plus a Logs Insights widget over the runtime log group grouped by cacheStatus. Console-only alarms on AvoidableMiss and WastedUsd Sums (stricter in prod, NOT_BREACHING on missing data so the PROMPT_CACHE_OBSERVABILITY_ENABLED kill switch stays quiet). No SNS — alerting infra is deliberately out of scope, matching kb-sync and scheduled-runs. Co-Authored-By: Claude Fable 5 * feat(admin-costs): add per-session cost-anatomy drill-down page Consumes GET /admin/costs/sessions/{id}/calls (backend PR #697): - SessionCostAnatomy / SessionCallRow / PrefixFingerprints models + CacheStatus union - AdminCostHttpService.getSessionCostAnatomy with URL-encoded session id - New /admin/costs/sessions/:id route (lazy, component input binding) - Drill-down page: summary rollups (total cost, cache efficiency incl. null, avoidable misses, wasted USD, cache read/write tokens), chronological calls table with color-coded cacheStatus badges, and prefix-fingerprint diffing that flags which hash (tools/system/history) flipped vs the previous fingerprinted call — the cache-buster diagnosis on miss_avoidable rows. Expandable rows show full hashes + messageCount; 404 renders a no-cost-rows empty state. - Session-id lookup form on the Cost Analytics dashboard as the entry point - Vitest specs for the diff util, HTTP method, and page states Co-Authored-By: Claude Fable 5 * fix(observability): don't flag first cache write after below-threshold calls as miss_avoidable When every prior call in a session was uncached (prompt below the model's minimum cacheable prefix, e.g. ~4096 tokens on Claude Haiku 4.5), the first call that crosses the threshold does cacheWrite>0/cacheRead=0 and was classified miss_avoidable — inflating the AvoidableMiss and WastedUsd EMF metrics and the admin Session Cost Anatomy page. Verified live in dev-ai session 9a1f25b2 (calls 1-5 uncached at 3.5-4k tokens, call 6 falsely flagged with write=4122/read=0). classify_cache_status now takes the previous call's cached-prefix token total: when the immediately preceding call had zero cache activity there was no entry to read from, so the write is classified first_write (the expected initial population) and excluded from waste pricing. Unknown (None) keeps the previous behavior. Co-Authored-By: Claude Fable 5 * Set S3_USER_FILES_BUCKET_NAME on inference-api runtime The AgentCore Runtime env block set DYNAMODB_USER_FILES_TABLE_NAME but not S3_USER_FILES_BUCKET_NAME, so the Word-document tools' _user_files_bucket() fell back to the literal 'user-files' default and PutObject failed with AccessDenied (and earlier PermanentRedirect against that unrelated bucket). The runtime role's UserFilesBucketAccess already grants Get/Put/Delete/List on the real bucket; this just points the runtime at it. tsc build passes. * Fail loudly when Word doc storage bucket is unconfigured _user_files_bucket() previously defaulted to a literal 'user-files' bucket when S3_USER_FILES_BUCKET_NAME was unset, which surfaced a missing runtime env var as a confusing S3 PermanentRedirect/AccessDenied. It now raises _StorageNotConfiguredError, and create/modify/read short-circuit before the Code Interpreter run with a clear 'storage is not configured' message. * Release/1.9.0 Feature release making Bedrock prompt-cache economics stable and measurable. Fixes three cache-busting defects in the model-call path (per-turn history mutation via the sliding truncation window, nondeterministic skill ordering, single fragile cachePoint) and adds end-to-end observability: prefix fingerprints + cacheStatus classification on every cost row, the admin Session Cost Anatomy page, per-call EMF metrics, and a CloudWatch dashboard with alarms. Also fixes chat-input textarea sizing and points the runtime's Word-document tools at the real user-files bucket. - Prompt-cache observability layer with PROMPT_CACHE_OBSERVABILITY_ENABLED kill switch (#697) - Admin Session Cost Anatomy drill-down at /admin/costs/sessions/:id (#700) - Byte-stable restored history, deterministic skill ordering, 3 cachePoints (#697) - first_write vs miss_avoidable classification fix after below-threshold calls (#701) - PromptCacheObservabilityConstruct dashboard + alarms (#699) - S3_USER_FILES_BUCKET_NAME on the inference-api Runtime + fail-loud storage guard (#702, #706) - Chat input textarea scroll/clamp/reset fixes (#696) Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Opus 4.8 Co-authored-by: Colin Smith <7762103+colinmxs@users.noreply.github.com> Co-authored-by: derrickfink Co-authored-by: Derrick Fink Co-authored-by: Roman Meredith Co-authored-by: Roman meredith <48036775+ramenNoodles1998@users.noreply.github.com> --- CHANGELOG.md | 31 ++ CLAUDE.MD | 3 + README.md | 4 +- RELEASE_NOTES.md | 66 +++ VERSION | 2 +- backend/pyproject.toml | 2 +- .../builtin_tools/word_document_tool.py | 33 +- backend/src/agents/main_agent/base_agent.py | 11 + .../src/agents/main_agent/config/constants.py | 3 + .../agents/main_agent/core/agent_factory.py | 19 +- .../agents/main_agent/core/model_config.py | 52 +- .../main_agent/session/compaction_models.py | 22 +- .../main_agent/session/hooks/__init__.py | 2 + .../session/hooks/prefix_fingerprint.py | 103 ++++ .../session/tests/test_compaction.py | 42 +- .../tests/test_compaction_integration.py | 7 - .../session/turn_based_session_manager.py | 130 ++++- .../main_agent/skills/strands_mapping.py | 5 + .../streaming/stream_coordinator.py | 47 +- .../src/apis/app_api/admin/costs/models.py | 49 ++ .../src/apis/app_api/admin/costs/routes.py | 55 ++ .../src/apis/app_api/admin/costs/service.py | 96 ++++ .../src/apis/shared/observability/__init__.py | 25 + backend/src/apis/shared/observability/emf.py | 89 ++++ .../apis/shared/observability/prompt_cache.py | 169 +++++++ backend/src/apis/shared/rbac/service.py | 10 +- backend/src/apis/shared/sessions/metadata.py | 228 ++++++++- backend/src/apis/shared/skills/repository.py | 8 +- .../apis/shared/storage/dynamodb_storage.py | 48 ++ .../core/test_bedrock_cache_points.py | 217 ++++++++ .../session/test_compaction_models.py | 12 + .../session/test_compaction_stability.py | 342 +++++++++++++ .../session/test_prefix_fingerprint_hook.py | 126 +++++ .../test_turn_based_session_manager.py | 63 +-- .../main_agent/skills/test_strands_mapping.py | 22 + .../test_prompt_cache_determinism.py | 173 +++++++ .../shared/test_metadata_cache_derivation.py | 245 +++++++++ .../shared/test_prompt_cache_observability.py | 235 +++++++++ .../tests/shared/test_skills_repository.py | 25 + backend/uv.lock | 2 +- docs/kaizen/review-queue.md | 14 + frontend/ai.client/package-lock.json | 4 +- frontend/ai.client/package.json | 2 +- .../ai.client/src/app/admin/admin.routes.ts | 4 + .../src/app/admin/costs/admin-costs.page.ts | 55 +- .../admin/costs/models/admin-cost.models.ts | 53 ++ .../pages/session-cost-anatomy.page.spec.ts | 160 ++++++ .../costs/pages/session-cost-anatomy.page.ts | 469 ++++++++++++++++++ .../pages/session-cost-anatomy.util.spec.ts | 98 ++++ .../costs/pages/session-cost-anatomy.util.ts | 62 +++ .../services/admin-cost-http.service.spec.ts | 29 ++ .../costs/services/admin-cost-http.service.ts | 12 + .../chat-input/chat-input.component.css | 22 - .../chat-input/chat-input.component.html | 2 +- .../chat-input/chat-input.component.ts | 33 +- .../inference-agentcore-construct.ts | 6 + .../prompt-cache-observability-construct.ts | 198 ++++++++ infrastructure/lib/platform-stack.ts | 12 + infrastructure/package-lock.json | 4 +- infrastructure/package.json | 2 +- .../test/prompt-cache-observability.test.ts | 86 ++++ 61 files changed, 3987 insertions(+), 163 deletions(-) create mode 100644 backend/src/agents/main_agent/session/hooks/prefix_fingerprint.py create mode 100644 backend/src/apis/shared/observability/__init__.py create mode 100644 backend/src/apis/shared/observability/emf.py create mode 100644 backend/src/apis/shared/observability/prompt_cache.py create mode 100644 backend/tests/agents/main_agent/core/test_bedrock_cache_points.py create mode 100644 backend/tests/agents/main_agent/session/test_compaction_stability.py create mode 100644 backend/tests/agents/main_agent/session/test_prefix_fingerprint_hook.py create mode 100644 backend/tests/agents/main_agent/test_prompt_cache_determinism.py create mode 100644 backend/tests/shared/test_metadata_cache_derivation.py create mode 100644 backend/tests/shared/test_prompt_cache_observability.py create mode 100644 frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.spec.ts create mode 100644 frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.ts create mode 100644 frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.spec.ts create mode 100644 frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.ts create mode 100644 infrastructure/lib/constructs/observability/prompt-cache-observability-construct.ts create mode 100644 infrastructure/test/prompt-cache-observability.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b2dd4383a..c6b30c72f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,37 @@ All notable changes to this project are documented in this file. Format follows For narrative release notes written for operators and product owners, see [RELEASE_NOTES.md](RELEASE_NOTES.md). +## [1.9.0] - 2026-07-20 + +Feature release making **Bedrock prompt-cache economics stable and measurable**. Three cache-busting defects in the model-call path are fixed (per-turn history mutation, nondeterministic skill ordering, single-cachePoint fragility), and a new observability layer makes every model call's cache behavior diagnosable: prefix fingerprints and a `cacheStatus` classification on each cost row, an admin Session Cost Anatomy drill-down page, CloudWatch EMF metrics, and a dashboard with alarms. Also fixes chat-input textarea sizing and points the deployed runtime's Word-document tools at the real user-files bucket. Requires a CDK deploy (new dashboard construct + one runtime env var); no data migration. + +### 🚀 Added + +- Prompt-cache observability per model call — `PrefixFingerprintHook` hashes the three cacheable prefix components (toolConfig, effective system prompt, message history) onto each cost row; write-time `cacheStatus` classification (`first_write | hit | miss_ttl_expired | miss_avoidable | uncached`) with `wastedUsd` for avoidable misses priced from the row's own pricing snapshot; session-row cache rollups (`totalCacheReadTokens`, `totalCacheWriteTokens`, `avoidableMissCount`, `wastedUsd`); `GET /admin/costs/sessions/{sessionId}/calls` admin endpoint; and per-call CloudWatch EMF metrics. Kill switch `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` (default ON) (#697) +- Admin **Session Cost Anatomy** page at `/admin/costs/sessions/:id` — chronological per-call table with color-coded `cacheStatus` badges and prefix-fingerprint diffing that names which hash (tools/system/history) flipped versus the previous call, plus session-level cache summary and a session-id lookup form on the Cost Analytics dashboard (#700) + +### ⚡ Performance + +- Restored conversation history is now byte-stable between turns — tool-content truncation is driven by a persisted `truncation_anchor` in compaction state instead of a sliding window that re-mutated history every turn, which forced a full prompt-cache prefix re-write (~$2.50/MTok on a 35k–150k-token prefix) nearly every turn (#697) +- Skill records reach the `` system-prompt block in deterministic order (sorted at the repository, RBAC-union, and injection layers), removing a per-turn prompt-cache invalidator (#697) +- Model calls now carry 3 of Bedrock's max-4 cachePoints (toolConfig tail, system tail, last-user-message) so a message-level cache miss still reads the stable tools+system prefix from cache instead of re-writing everything; the added points are gated to models that support them (#697) + +### 🐛 Fixed + +- Chat input textarea is scrollable once content exceeds its max height, clamps its growth, and resets to its base height after sending (#696) +- The first cache write after a run of below-threshold (uncached) calls is classified `first_write`, not `miss_avoidable` — it no longer inflates the AvoidableMiss/WastedUsd metrics or the Session Cost Anatomy page (#701) +- Word-document tools on the deployed runtime save to the real user-files bucket — `S3_USER_FILES_BUCKET_NAME` is now set on the inference-api Runtime, where its absence made the tools fall back to a literal `user-files` bucket and fail with `AccessDenied` (#702) +- Word-document tools fail fast with a clear "storage is not configured" message when the bucket env var is unset, instead of surfacing a confusing S3 `PermanentRedirect`/`AccessDenied` mid-run (#706) + +### 🏗️ Infrastructure + +- `PromptCacheObservabilityConstruct` (`lib/constructs/observability/`) — CloudWatch dashboard over the `AgentCoreStack/PromptCache` EMF metrics (cache read/write tokens, cache-efficiency expression, AvoidableMiss, WastedUsd) with a Logs Insights widget grouped by `cacheStatus`, plus console-only alarms on AvoidableMiss and WastedUsd (stricter in prod; `NOT_BREACHING` on missing data so the kill switch stays quiet) (#699) +- `S3_USER_FILES_BUCKET_NAME` added to the inference-api AgentCore Runtime environment (#702) + +### 📚 Docs + +- CLAUDE.md — prompt-cache determinism/byte-stability contract, the fingerprint-based cache-miss debugging recipe, and a token-cost-effectiveness design tenet (#697, #698) + ## [1.8.0] - 2026-07-19 Feature release delivering **Skills v2** — skills are redesigned from tool-binding containers into pure, portable knowledge bundles that Agents load on demand, and for the first time **any signed-in user can author their own**. Skills ship enabled by default. Also completes the session-metadata static-sort-key migration (issue #175 Phases 2–3) with an operator backfill and a GSI-only read contraction, and consolidates the two artifact tool-catalog rows into a single "Artifacts" toggle. **Three backfill scripts must be run per environment** — see [RELEASE_NOTES.md](RELEASE_NOTES.md) deployment notes. No new AWS resources; a CDK deploy is needed only to set the `SKILLS_ENABLED` env var explicitly. diff --git a/CLAUDE.MD b/CLAUDE.MD index 8e1af5c83..7d74fa792 100644 --- a/CLAUDE.MD +++ b/CLAUDE.MD @@ -39,6 +39,8 @@ npx cdk deploy {prefix}-PlatformStack - **Admin endpoints** go under `/admin//`, user-facing under `//` - **Errors stream as assistant messages** via SSE (not HTTP error codes) - **Signal-based state** throughout frontend (`signal()`, `computed()`) +- **Prompt-cache stability is a contract.** Bedrock prompt caching is exact-prefix-match: any list that reaches the system prompt or `toolConfig` (skills, tools, models, MCP tool listings) must be deterministically ordered at its source, and restored conversation history must be byte-stable between compaction-state changes (see the truncation anchor in `TurnBasedSessionManager`). An order flip or history mutation between turns silently re-writes a 30k–150k-token prefix at the $2.50/MTok cache-write premium. `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` disables the observability layer (fingerprint hook, cacheStatus derivation, EMF metrics) — the caching itself stays on +- **Token cost effectiveness is a design tenet — engineer against waste, not against context.** Before merging a change that touches the model call path, answer: what does this add to the prompt, on every turn, for the life of every session? (1) Anything in the cacheable prefix (system prompt, `toolConfig`, restored history) must be deterministic and append-only — see the prompt-cache contract above. (2) Per-turn payloads (tool results, MCP responses, retrieved documents) should be bounded or offloaded, never unbounded pass-through. (3) Don't guess — verify: `cacheStatus` + fingerprint hashes on the session's `C#` rows, `GET /admin/costs/sessions/{id}/calls`, and the `AgentCoreStack/PromptCache` EMF metrics exist to prove a change's cost impact. The balance: "cost effective" means eliminating waste (avoidable cache re-writes, duplicated context, oversized payloads) — it never means stripping context the model needs for a quality answer. When cost and answer quality genuinely conflict, quality wins; look for the cheaper path to the *same* quality, not a cheaper answer - **All dependencies use exact version pins** — no `^`, `~`, or `>=` - **Never install packages without explicit user approval** @@ -118,6 +120,7 @@ Do not reintroduce a Bearer-only `Depends(...)` on any user-facing route. If you - **Tool not appearing:** Check `__init__.py` export, RBAC permissions, `enabled_tools`, ToolRegistry - **Session not persisting:** Check AgentCore Memory config, session_id, TurnBasedSessionManager flush - **SSE stream disconnecting:** Check 600s timeout, client connection, quota exceeded events +- **Cost spike / cache miss:** Check `cacheStatus` + `toolConfigHash`/`systemPromptHash`/`historyHash` on the session's `C#` rows in sessions-metadata (or `GET /admin/costs/sessions/{id}/calls`) — the hash that changed between consecutive calls names the cache-buster ## Coding Standards diff --git a/README.md b/README.md index 5b9906bb8..c014c9a75 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **An open-source, production-ready Generative AI platform for institutions** *Built by Boise State University, designed for everyone.* -[![Release](https://img.shields.io/badge/Release-v1.8.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) +[![Release](https://img.shields.io/badge/Release-v1.9.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) [![Nightly](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml) ![Python](https://img.shields.io/badge/Python-3.13+-3776AB?style=flat&logo=python&logoColor=white) @@ -296,7 +296,7 @@ agentcore-public-stack/ See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full changelog, including new features, bug fixes, platform upgrades, and deployment notes for each release. -**Current release:** v1.8.0 +**Current release:** v1.9.0 --- diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 5c9db597f..dd55d3682 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,3 +1,69 @@ +# Release Notes — v1.9.0 + +**Release Date:** July 20, 2026 +**Previous Release:** v1.8.0 (July 19, 2026) + +--- + +> 🏗️ **CDK deploy required this release** — a new CloudWatch dashboard construct and one new env var on the inference-api Runtime. No data migration, no backfills, no dependency changes. Standard order: `platform.yml` → `backend.yml` → `frontend-deploy.yml`. + +--- + +## Highlights + +v1.9.0 makes **Bedrock prompt-cache spend stable and measurable**. A production conversation audit found 75% of one session's cost was avoidable cache re-writes; this release fixes the three defects behind that waste — restored history that mutated every turn, skills injected in nondeterministic order, and a single fragile cachePoint — and adds the observability to prove it and catch regressions: every model call now records prefix fingerprints and a `cacheStatus` verdict, admins get a per-session **Cost Anatomy** drill-down page, and operators get a CloudWatch dashboard with alarms on avoidable waste. Smaller fixes: the chat input textarea scrolls and resets correctly, and Word-document saves work on the deployed runtime (the container wasn't told which S3 bucket to use). + +## Prompt-cache stability — stop paying for avoidable re-writes + +Bedrock prompt caching is exact-prefix-match: if any byte of the cached prefix changes between turns, the whole prefix re-writes at the $2.50/MTok cache-write premium instead of reading at the ~$0.30/MTok cache-read rate. On a typical 35k–150k-token session prefix, one silent cache-buster costs more per turn than most turns' actual work. Three were found and fixed (#697): + +- **Restored history is now byte-stable.** Tool-content truncation ran on every session restore behind a sliding protected-turns window, so each new turn re-mutated the turn that had just aged past the window — invalidating the cache nearly every turn. Truncation is now driven by a persisted `truncation_anchor` in the session's compaction state: it advances only when the compaction checkpoint advances (where the slice already pays a single re-write) or opportunistically when the cache TTL (default 300s, `AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS`) has lapsed since the previous turn and the cache entry is dead anyway — so pending truncations apply for free. +- **Skills inject in deterministic order.** Skill records reached the `` system-prompt block in whatever order DynamoDB `batch_get_item` and Python set iteration produced, changing the system prompt between turns of the same session. Ordering is now sorted at three layers: the skills repository, the RBAC grant-union resolver (which returned `list(set)` — order varies per process via hash randomization), and the injection point itself. +- **Three cachePoints instead of one.** The auto strategy places a single message-level cachePoint; when its lookup misses (one proven mode: a wide parallel tool fan-out pushes the previous checkpoint past Anthropic's ~20-block cache lookback), *nothing* was read and the entire prefix re-wrote. Requests now carry 3 of Bedrock's max-4 cachePoints — toolConfig tail, system-prompt tail, and the existing last-user-message point — so a message-level miss still reads the stable tools+system prefix from cache. The added points are gated on `ModelConfig.bedrock_cache_points_supported()` since non-Anthropic models reject them. + +## Prompt-cache observability — every model call explains its cache behavior + +Diagnosing the waste above originally took hours of manual forensics against raw DynamoDB cost rows. That whole class of investigation is now a column diff (#697, #700, #699, #701). + +### Backend + +- `PrefixFingerprintHook` (a Strands `BeforeModelCallEvent` hook) hashes the three cacheable prefix components per model call — toolConfig (order-sensitive canonical JSON), the effective system prompt captured *after* AgentSkills injection, and message history excluding the newest message — and the stream coordinator persists them on the turn's cost rows. When a cache miss happens, the hash that changed between consecutive calls names the cache-buster. +- Each cost row gets a write-time `cacheStatus` — `first_write`, `hit`, `miss_ttl_expired`, `miss_avoidable`, or `uncached` — derived against the session's previous cost row, plus `wastedUsd` for avoidable misses priced at the cache-write premium over cache-read from the row's own pricing snapshot. Turn rows now write sequentially so each call classifies against its true predecessor. A follow-up fix (#701) classifies the first write after a run of below-threshold calls as `first_write` rather than `miss_avoidable`, so short-prompt sessions don't inflate the waste metrics. +- Session rows carry rollups next to `totalCost` — `totalCacheReadTokens`, `totalCacheWriteTokens`, `avoidableMissCount`, `wastedUsd` — so lists and admin views get a cache-efficiency ratio without scanning cost rows. +- `GET /admin/costs/sessions/{sessionId}/calls` (admin-only) returns the chronological per-call rows with token splits, cost, `cacheStatus`, and fingerprints, plus a session-level cache summary. +- Everything derived is behind `PROMPT_CACHE_OBSERVABILITY_ENABLED` (default ON, `=false` to disable the hook, the classification's extra GSI read, and EMF emission). Raw cache read/write token rollups are unaffected — they're usage passthrough, not derived. + +### Frontend + +- New **Session Cost Anatomy** page at `/admin/costs/sessions/:id`: summary tiles (total cost, cache efficiency, avoidable misses, wasted USD, cache read/write tokens) over a chronological calls table with color-coded `cacheStatus` badges. Fingerprint diffing flags which hash — tools, system, or history — flipped versus the previous fingerprinted call, which is the diagnosis on any `miss_avoidable` row. Expandable rows show full hashes and message counts; a session-id lookup form on the Cost Analytics dashboard is the entry point. + +### Infrastructure + +- `PromptCacheObservabilityConstruct` (new `lib/constructs/observability/` area, composed into `PlatformStack`) builds a CloudWatch dashboard over the `AgentCoreStack/PromptCache` EMF namespace both APIs emit into: cache read/write token trends, a cache-efficiency MathExpression, AvoidableMiss and WastedUsd, and a Logs Insights widget grouped by `cacheStatus`. Console-only alarms on AvoidableMiss and WastedUsd Sums (stricter thresholds in prod, `NOT_BREACHING` on missing data so the kill switch keeps them quiet). Deliberately no SNS — alerting infra remains out of scope, matching kb-sync and scheduled runs. + +### Test Coverage + +~1,800 lines of new tests: fingerprint/classification unit tests (including the below-threshold regression), cachePoint position and budget assertions, forced-order skill-sorting regressions, CDK construct assertions, and Vitest specs for the anatomy page, diff util, and HTTP service. + +## 🐛 Bug fixes + +- **The chat input became unusable on long prompts.** The textarea carried `overflow-hidden` with unclamped height growth, so past its 200px cap the content could neither be seen nor scrolled — and after sending, the box stayed expanded. Growth is now clamped with scrolling enabled past the cap, and the input resets to its base height on submit (#696) +- **Word-document saves failed on the deployed runtime with `AccessDenied`.** The AgentCore Runtime env set the user-files *table* name but not `S3_USER_FILES_BUCKET_NAME`, so the Word tools fell back to a literal `user-files` bucket the role has no access to. The env var is now set on the Runtime (#702), and the tools fail fast with a clear "storage is not configured" message — before spending a Code Interpreter run — if the variable is ever missing again (#706) + +## 🏗️ Infrastructure + +- New CloudWatch dashboard + alarms construct (see spotlight above) — CloudWatch-console resources only, no SNS, no new IAM of note. +- `S3_USER_FILES_BUCKET_NAME` on the inference-api Runtime environment; the role's existing `UserFilesBucketAccess` grant already covers the bucket (#702) +- New env var `PROMPT_CACHE_OBSERVABILITY_ENABLED` on app-api and inference-api (default ON; set `=false` per environment to disable the observability layer — caching itself stays on). + +## 🚀 Deployment notes + +Standard order, and this release uses all three: **`platform.yml` first** (the dashboard construct and the Runtime env var are CDK changes; the runtime picks up its current image via SSM, so the infra deploy is safe on its own) → `backend.yml` → `frontend-deploy.yml`. No backfills, no data migration, no dependency changes. + +After deploy, the **PromptCache dashboard** appears in the CloudWatch console. Expect `first_write` rows at the start of sessions and after idle gaps — only `miss_avoidable` indicates regression. The observability layer is per-call metadata; if it ever needs to be silenced in an environment, set `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` and redeploy that service — the alarms go quiet on missing data by design. + +--- + # Release Notes — v1.8.0 **Release Date:** July 19, 2026 diff --git a/VERSION b/VERSION index 27f9cd322..f8e233b27 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.8.0 +1.9.0 diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 57f66059b..5ccd017eb 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agentcore-stack" -version = "1.8.0" +version = "1.9.0" requires-python = ">=3.10" description = "Multi-agent conversational AI system with AWS Bedrock AgentCore" readme = "README.md" diff --git a/backend/src/agents/builtin_tools/word_document_tool.py b/backend/src/agents/builtin_tools/word_document_tool.py index 3e517bb4d..e336fb7ad 100644 --- a/backend/src/agents/builtin_tools/word_document_tool.py +++ b/backend/src/agents/builtin_tools/word_document_tool.py @@ -69,6 +69,10 @@ class _DocGenError(Exception): """Raised when Code Interpreter fails to run the document code.""" +class _StorageNotConfiguredError(Exception): + """Raised when the user-files S3 bucket is not configured for the runtime.""" + + def _region() -> str: return ( os.environ.get("AWS_REGION") @@ -190,7 +194,18 @@ def _s3(): def _user_files_bucket() -> str: - return os.environ.get("S3_USER_FILES_BUCKET_NAME", "user-files") + # Fail loudly rather than silently targeting a literal "user-files" bucket + # the runtime has no access to. That default misreported a missing env var + # as an S3 PermanentRedirect / AccessDenied and cost real debugging time. + # The runtime env is wired in infrastructure's + # inference-agentcore-construct.ts (S3_USER_FILES_BUCKET_NAME). + bucket = os.environ.get("S3_USER_FILES_BUCKET_NAME") + if not bucket: + raise _StorageNotConfiguredError( + "S3_USER_FILES_BUCKET_NAME is not set; the runtime cannot store or " + "retrieve Word documents." + ) + return bucket # --------------------------------------------------------------------------- @@ -512,6 +527,16 @@ def _error(text: str) -> Dict[str, Any]: "not found in the environment or Parameter Store." ) +_NO_STORAGE_MESSAGE = ( + "❌ Word document storage is not configured " + "(S3_USER_FILES_BUCKET_NAME is not set on the runtime)." +) + + +def _storage_configured() -> bool: + """True when the user-files bucket env var is set.""" + return bool(os.environ.get("S3_USER_FILES_BUCKET_NAME")) + # --------------------------------------------------------------------------- # Tool factories @@ -577,6 +602,8 @@ async def create_word_document( code_interpreter_id = _get_code_interpreter_id() if not code_interpreter_id: return _error(_NO_CI_MESSAGE) + if not _storage_configured(): + return _error(_NO_STORAGE_MESSAGE) try: file_bytes = await asyncio.to_thread( @@ -642,6 +669,8 @@ async def modify_word_document( code_interpreter_id = _get_code_interpreter_id() if not code_interpreter_id: return _error(_NO_CI_MESSAGE) + if not _storage_configured(): + return _error(_NO_STORAGE_MESSAGE) source = await _find_word_document(user_id, session_id, document_name) if source is None: @@ -754,6 +783,8 @@ async def read_word_document(document_name: str) -> Dict[str, Any]: code_interpreter_id = _get_code_interpreter_id() if not code_interpreter_id: return _error(_NO_CI_MESSAGE) + if not _storage_configured(): + return _error(_NO_STORAGE_MESSAGE) source = await _find_word_document(user_id, session_id, document_name) if source is None: diff --git a/backend/src/agents/main_agent/base_agent.py b/backend/src/agents/main_agent/base_agent.py index acdc7b133..58210b783 100644 --- a/backend/src/agents/main_agent/base_agent.py +++ b/backend/src/agents/main_agent/base_agent.py @@ -17,6 +17,7 @@ OAuthConsentHook, MCPExternalApprovalHook, ContextAttributionHook, + PrefixFingerprintHook, ) from agents.main_agent.tools import ( create_default_registry, @@ -297,6 +298,16 @@ def _create_hooks(self) -> List: # final metadata SSE event. hooks.append(ContextAttributionHook()) + # Per-model-call prompt-cache prefix fingerprints (toolConfig / + # system prompt / history hashes). Best-effort; the stream + # coordinator persists them on each call's metadata row so avoidable + # cache misses are diagnosable from stored data. + # PROMPT_CACHE_OBSERVABILITY_ENABLED=false disables the layer. + from apis.shared.observability import prompt_cache_observability_enabled + + if prompt_cache_observability_enabled(): + hooks.append(PrefixFingerprintHook()) + return hooks def _build_mcp_external_approval_hook(self) -> MCPExternalApprovalHook: diff --git a/backend/src/agents/main_agent/config/constants.py b/backend/src/agents/main_agent/config/constants.py index c817f463f..48ceeb481 100644 --- a/backend/src/agents/main_agent/config/constants.py +++ b/backend/src/agents/main_agent/config/constants.py @@ -27,6 +27,7 @@ class EnvVars: COMPACTION_TOKEN_THRESHOLD = "AGENTCORE_MEMORY_COMPACTION_TOKEN_THRESHOLD" COMPACTION_PROTECTED_TURNS = "AGENTCORE_MEMORY_COMPACTION_PROTECTED_TURNS" COMPACTION_MAX_TOOL_CONTENT_LENGTH = "AGENTCORE_MEMORY_COMPACTION_MAX_TOOL_CONTENT_LENGTH" + COMPACTION_CACHE_TTL_SECONDS = "AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS" # --- Restored-history repair --- # Kill switch for the restore-time tool-pairing/alternation repair @@ -98,6 +99,8 @@ class Defaults: COMPACTION_TOKEN_THRESHOLD = 100_000 COMPACTION_PROTECTED_TURNS = 3 COMPACTION_MAX_TOOL_CONTENT_LENGTH = 500 + # Bedrock prompt-cache TTL (seconds); see CompactionConfig.cache_ttl_seconds + COMPACTION_CACHE_TTL_SECONDS = 300 # --- DynamoDB Tables --- DYNAMODB_QUOTA_TABLE = "UserQuotas" diff --git a/backend/src/agents/main_agent/core/agent_factory.py b/backend/src/agents/main_agent/core/agent_factory.py index 6d9e99852..95a57b89b 100644 --- a/backend/src/agents/main_agent/core/agent_factory.py +++ b/backend/src/agents/main_agent/core/agent_factory.py @@ -196,12 +196,29 @@ def create_agent( f"({model_config.retry_config.sdk_initial_delay}s-{model_config.retry_config.sdk_max_delay}s backoff)" ) + # Bedrock prompt caching: give the system prompt its own cachePoint by + # passing it as a SystemContentBlock list with a trailing cachePoint + # (the cache_prompt model-config key is deprecated). Together with + # cache_tools (set in to_bedrock_config) this keeps the stable + # system+tools prefix readable from cache even when the auto-placed + # message-level cache point misses — see the cachePoint budget comment + # in ModelConfig.to_bedrock_config. Strands' auto strategy strips only + # message-level cachePoints, never system ones. Agent.system_prompt + # remains the plain string (split_system_prompt concatenates the text + # blocks), so hashing/attribution/voice consumers are unaffected. + agent_system_prompt: Any = system_prompt + if system_prompt and model_config.bedrock_cache_points_supported(): + agent_system_prompt = [ + {"text": system_prompt}, + {"cachePoint": {"type": "default"}}, + ] + # Create agent with session manager, hooks, and system prompt # Use SequentialToolExecutor to prevent concurrent browser operations # This prevents "Failed to start and initialize Playwright" errors with NovaAct agent = Agent( model=model, - system_prompt=system_prompt, + system_prompt=agent_system_prompt, tools=tools, tool_executor=SequentialToolExecutor(), session_manager=session_manager, diff --git a/backend/src/agents/main_agent/core/model_config.py b/backend/src/agents/main_agent/core/model_config.py index af99f9acf..422ea57f5 100644 --- a/backend/src/agents/main_agent/core/model_config.py +++ b/backend/src/agents/main_agent/core/model_config.py @@ -311,6 +311,23 @@ def get_provider(self) -> ModelProvider: # Default to configured provider return self.provider + def bedrock_cache_points_supported(self) -> bool: + """Whether explicit Bedrock cachePoints (tools/system) may be sent. + + Mirrors Strands' ``BedrockModel._cache_strategy`` predicate: Anthropic + models are the only Bedrock family with prompt-cache support. Auto + (message-level) caching no-ops safely on other models, but explicit + tools/system cachePoints would be sent verbatim and rejected with a + ValidationException — so both are gated here, alongside + ``caching_enabled``, on the Bedrock provider path. + """ + model_lower = self.model_id.lower() + return ( + self.caching_enabled + and self.get_provider() == ModelProvider.BEDROCK + and ("claude" in model_lower or "anthropic" in model_lower) + ) + def to_bedrock_config(self) -> Dict[str, Any]: """Convert to BedrockModel kwargs, translating canonical inference params.""" config: Dict[str, Any] = {"model_id": self.model_id} @@ -330,11 +347,34 @@ def to_bedrock_config(self) -> Dict[str, Any]: # unconditionally on the Bedrock path. config["use_native_token_count"] = True - # Bedrock prompt caching. CacheConfig(strategy="auto") lets Strands - # place cache points per-model: for a model that supports automatic - # caching it injects a cachePoint on the system/tools/last-user blocks; - # for one that doesn't it logs a warning and no-ops, so this is safe to - # set whenever caching is enabled. Requires strands-agents>=1.48.0: a + # Bedrock prompt caching — three cachePoints per request (Bedrock + # allows max 4; nothing else in this codebase adds one, see the + # position test in tests/agents/main_agent/core/test_bedrock_cache_points.py): + # + # 1. toolConfig tail — cache_tools="default" (_build_tools_cache_point) + # 2. system tail — SystemContentBlock list built by + # AgentFactory.create_agent (the deprecated + # cache_prompt config key is NOT used) + # 3. last user message — CacheConfig(strategy="auto"), which places + # exactly ONE message-level point and strips + # any others (_inject_cache_point). It does + # not touch the system/tools points. + # + # The tools+system points make a message-level lookup miss cost a + # cache READ of the stable prefix instead of a full re-write at + # $2.5/MTok (write premium). One proven miss mode is structural: + # Anthropic's cache lookback checks only ~20 content blocks behind + # the breakpoint, so a wide parallel tool fan-out (e.g. 18 parallel + # calls = ~38 new blocks) pushes the previous checkpoint out of range + # and forces a full re-write (prod session aecd387d: cacheRead=0, + # cacheWrite=134k mid-turn). With separate tools/system points the + # ~28k-token static prefix still reads from cache on those turns. + # + # For a model whose id Strands doesn't recognize as cache-capable, + # auto strategy logs a warning and no-ops — but cache_tools and a + # system cachePoint are sent unconditionally once configured, so both + # are gated on bedrock_cache_points_supported() (the same predicate + # Strands' auto mode uses). Requires strands-agents>=1.48.0: a # cachePoint trailing a non-PDF `document` attachment is rejected by # Bedrock's Anthropic adapter with "ValidationException ... # content.N.type: Field required" (agent force-stop on any turn with a @@ -347,6 +387,8 @@ def to_bedrock_config(self) -> Dict[str, Any]: if self.caching_enabled: from strands.models import CacheConfig config["cache_config"] = CacheConfig(strategy="auto") + if self.bedrock_cache_points_supported(): + config["cache_tools"] = "default" if self.retry_config: from botocore.config import Config as BotocoreConfig diff --git a/backend/src/agents/main_agent/session/compaction_models.py b/backend/src/agents/main_agent/session/compaction_models.py index a0fbc506c..3758094f3 100644 --- a/backend/src/agents/main_agent/session/compaction_models.py +++ b/backend/src/agents/main_agent/session/compaction_models.py @@ -29,6 +29,14 @@ class CompactionState: # compaction event in this session. Surfaced on session-metadata GET so # the frontend's end-of-conversation indicator survives a refresh. total_summarized_turns: int = 0 + # Absolute message index below which tool contents are truncated on + # restore. Bedrock prompt caching requires an exact prefix match, so + # truncation must be a pure function of persisted state — this anchor + # only moves when the checkpoint advances (the slice already forces a + # cache re-write) or when the prompt cache has already expired between + # turns (the re-write is free then). It must never be derived from a + # per-restore sliding window. + truncation_anchor: int = 0 def to_dict(self) -> Dict[str, Any]: """Convert to dictionary for DynamoDB storage.""" @@ -38,6 +46,7 @@ def to_dict(self) -> Dict[str, Any]: "lastInputTokens": self.last_input_tokens, "updatedAt": self.updated_at, "totalSummarizedTurns": self.total_summarized_turns, + "truncationAnchor": self.truncation_anchor, } @classmethod @@ -45,12 +54,17 @@ def from_dict(cls, data: Optional[Dict[str, Any]]) -> "CompactionState": """Create from DynamoDB item dictionary.""" if not data: return cls() + checkpoint = int(data.get("checkpoint", 0)) return cls( - checkpoint=int(data.get("checkpoint", 0)), + checkpoint=checkpoint, summary=data.get("summary"), last_input_tokens=int(data.get("lastInputTokens", 0)), updated_at=data.get("updatedAt"), total_summarized_turns=int(data.get("totalSummarizedTurns", 0)), + # Legacy records predate the anchor: default it to the checkpoint + # so nothing retained by the slice is truncated (byte-stable from + # the first restore under the anchor design). + truncation_anchor=int(data.get("truncationAnchor", checkpoint)), ) @@ -83,6 +97,11 @@ class CompactionConfig: token_threshold: int = 100_000 # Trigger checkpoint when exceeded protected_turns: int = 3 # Recent turns to protect from truncation max_tool_content_length: int = 500 # Max chars before truncating tool output + # Bedrock prompt-cache TTL. When more than this many seconds have passed + # since the previous turn, the cache entry has already expired, so pending + # truncations can be applied without forcing an otherwise-avoidable + # prefix re-write. + cache_ttl_seconds: int = 300 @classmethod def from_env(cls) -> "CompactionConfig": @@ -92,4 +111,5 @@ def from_env(cls) -> "CompactionConfig": token_threshold=int(os.environ.get(EnvVars.COMPACTION_TOKEN_THRESHOLD, str(Defaults.COMPACTION_TOKEN_THRESHOLD))), protected_turns=int(os.environ.get(EnvVars.COMPACTION_PROTECTED_TURNS, str(Defaults.COMPACTION_PROTECTED_TURNS))), max_tool_content_length=int(os.environ.get(EnvVars.COMPACTION_MAX_TOOL_CONTENT_LENGTH, str(Defaults.COMPACTION_MAX_TOOL_CONTENT_LENGTH))), + cache_ttl_seconds=int(os.environ.get(EnvVars.COMPACTION_CACHE_TTL_SECONDS, str(Defaults.COMPACTION_CACHE_TTL_SECONDS))), ) diff --git a/backend/src/agents/main_agent/session/hooks/__init__.py b/backend/src/agents/main_agent/session/hooks/__init__.py index 9b9748444..017112a59 100644 --- a/backend/src/agents/main_agent/session/hooks/__init__.py +++ b/backend/src/agents/main_agent/session/hooks/__init__.py @@ -2,12 +2,14 @@ from agents.main_agent.session.hooks.context_attribution import ContextAttributionHook from agents.main_agent.session.hooks.oauth_consent import OAuthConsentHook +from agents.main_agent.session.hooks.prefix_fingerprint import PrefixFingerprintHook from agents.main_agent.session.hooks.stop import StopHook from agents.main_agent.session.hooks.tool_approval import MCPExternalApprovalHook __all__ = [ "ContextAttributionHook", "OAuthConsentHook", + "PrefixFingerprintHook", "StopHook", "MCPExternalApprovalHook", ] diff --git a/backend/src/agents/main_agent/session/hooks/prefix_fingerprint.py b/backend/src/agents/main_agent/session/hooks/prefix_fingerprint.py new file mode 100644 index 000000000..2d2b27155 --- /dev/null +++ b/backend/src/agents/main_agent/session/hooks/prefix_fingerprint.py @@ -0,0 +1,103 @@ +"""Hook that fingerprints the cacheable request prefix before each model call. + +Bedrock prompt caching is exact-prefix-match: toolConfig, then system prompt, +then message history must be byte-identical to the previous call for cached +tokens to be read. When a session shows ``cacheStatus=miss_avoidable`` on its +metadata rows, these three hashes — persisted per call alongside tokenUsage — +show *which* component diverged between consecutive calls, replacing hours of +raw-row forensics with a column diff. + +On every ``BeforeModelCallEvent`` the hook computes: + +- ``toolConfigHash``: canonical JSON of the tool specs the model will see, in + registration order (order-sensitive on purpose — order flips are a real + cache-buster). +- ``systemPromptHash``: the effective system prompt. Fires after plugin + injection (e.g. AgentSkills' ```` block lands on + ``BeforeInvocationEvent``), so nondeterministic skill ordering is visible + here. +- ``historyHash``: ``agent.messages`` excluding the newest message — the + prior-history prefix that must match the previous call's full history for + a cache hit. + +Fingerprints accumulate in a per-turn list on the agent (one entry per model +call; a tool-use turn makes several). The stream coordinator resets the list +at turn start and attaches entry N to the Nth assistant message's metadata +row. Best-effort: any failure is swallowed so fingerprinting can never break +a model call. +""" + +import logging +from typing import Any, Dict, List, Optional + +from strands.hooks import BeforeModelCallEvent, HookProvider, HookRegistry + +from apis.shared.observability import fingerprint_canonical_json, fingerprint_text + +logger = logging.getLogger(__name__) + +# Per-turn list of fingerprint dicts, stashed on the Strands agent instance. +_FINGERPRINTS_ATTR = "_prefix_fingerprints" + + +def reset_prefix_fingerprints(agent: Any) -> None: + """Clear the per-turn fingerprint list. Called at turn start.""" + setattr(agent, _FINGERPRINTS_ATTR, []) + + +def get_prefix_fingerprint(agent: Any, index: Optional[int] = None) -> Optional[Dict[str, Any]]: + """Return the fingerprint for the ``index``-th model call of this turn. + + ``index=None`` returns the latest entry (used by single-call persistence + paths like the interrupted-turn writer). Returns None when the hook never + fired (non-Bedrock paths, hook disabled, or index out of range). + """ + fingerprints: List[Dict[str, Any]] = getattr(agent, _FINGERPRINTS_ATTR, None) or [] + if not fingerprints: + return None + if index is None: + return fingerprints[-1] + if 0 <= index < len(fingerprints): + return fingerprints[index] + # More assistant messages than observed model calls (shouldn't happen) — + # better to attach nothing than a wrong fingerprint. + return None + + +class PrefixFingerprintHook(HookProvider): + """Compute toolConfig / system-prompt / history hashes per model call.""" + + def register_hooks(self, registry: HookRegistry, **kwargs: Any) -> None: + registry.add_callback(BeforeModelCallEvent, self._on_before_model_call) + + def _on_before_model_call(self, event: BeforeModelCallEvent) -> None: + try: + agent = event.agent + + tool_specs = agent.tool_registry.get_all_tool_specs() + system_prompt = getattr(agent, "system_prompt", None) + messages = list(getattr(agent, "messages", None) or []) + + # System prompt may be a plain string or structured content blocks + # (SystemContentBlock list, used to place cache points). + if isinstance(system_prompt, str) or system_prompt is None: + system_hash = fingerprint_text(system_prompt) + else: + system_hash = fingerprint_canonical_json(system_prompt) + + fingerprint = { + "toolConfigHash": fingerprint_canonical_json(tool_specs), + "systemPromptHash": system_hash, + # History *prefix*: everything except the newest message. For + # a cache hit this must equal the previous call's full history. + "historyHash": fingerprint_canonical_json(messages[:-1]), + "messageCount": len(messages), + } + + fingerprints = getattr(agent, _FINGERPRINTS_ATTR, None) + if fingerprints is None: + fingerprints = [] + setattr(agent, _FINGERPRINTS_ATTR, fingerprints) + fingerprints.append(fingerprint) + except Exception as e: # noqa: BLE001 - observability must never break a turn + logger.debug("Prefix fingerprinting skipped: %s", e) diff --git a/backend/src/agents/main_agent/session/tests/test_compaction.py b/backend/src/agents/main_agent/session/tests/test_compaction.py index 740369847..eaf7c5af9 100644 --- a/backend/src/agents/main_agent/session/tests/test_compaction.py +++ b/backend/src/agents/main_agent/session/tests/test_compaction.py @@ -19,32 +19,44 @@ def test_default_state(self): assert state.summary is None assert state.last_input_tokens == 0 assert state.updated_at is None + assert state.truncation_anchor == 0 def test_to_dict(self): state = CompactionState( checkpoint=10, summary="Test summary", last_input_tokens=50000, - updated_at="2025-01-15T10:00:00Z" + updated_at="2025-01-15T10:00:00Z", + truncation_anchor=14, ) d = state.to_dict() assert d["checkpoint"] == 10 assert d["summary"] == "Test summary" assert d["lastInputTokens"] == 50000 assert d["updatedAt"] == "2025-01-15T10:00:00Z" + assert d["truncationAnchor"] == 14 def test_from_dict(self): data = { "checkpoint": 5, "summary": "Previous context", "lastInputTokens": 75000, - "updatedAt": "2025-01-15T12:00:00Z" + "updatedAt": "2025-01-15T12:00:00Z", + "truncationAnchor": 9, } state = CompactionState.from_dict(data) assert state.checkpoint == 5 assert state.summary == "Previous context" assert state.last_input_tokens == 75000 assert state.updated_at == "2025-01-15T12:00:00Z" + assert state.truncation_anchor == 9 + + def test_from_dict_legacy_record_defaults_anchor_to_checkpoint(self): + """Records written before the anchor existed must not re-truncate + retained history: the anchor defaults to the checkpoint, so nothing + the slice keeps is mutated.""" + state = CompactionState.from_dict({"checkpoint": 7}) + assert state.truncation_anchor == 7 def test_from_dict_handles_none(self): state = CompactionState.from_dict(None) @@ -66,18 +78,21 @@ def test_default_config(self): assert config.token_threshold == 100_000 assert config.protected_turns == 3 assert config.max_tool_content_length == 500 + assert config.cache_ttl_seconds == 300 def test_from_env(self, monkeypatch): monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_ENABLED", "true") monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_TOKEN_THRESHOLD", "50000") monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_PROTECTED_TURNS", "3") monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_MAX_TOOL_CONTENT_LENGTH", "1000") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS", "600") config = CompactionConfig.from_env() assert config.enabled is True assert config.token_threshold == 50000 assert config.protected_turns == 3 assert config.max_tool_content_length == 1000 + assert config.cache_ttl_seconds == 600 class TestCompactionHelpers: @@ -159,29 +174,6 @@ def test_find_valid_cutoff_indices(self, mock_messages): # Should find indices 0, 4, 8 (the actual user questions, not tool results) assert valid_indices == [0, 4, 8] - def test_find_protected_indices(self, mock_messages): - """Test finding indices that should be protected from truncation""" - protected_turns = 2 - - # Find valid cutoff indices first - turn_start_indices = [] - for i, msg in enumerate(mock_messages): - if msg.get('role') == 'user': - content = msg.get('content', []) - is_tool_result = any( - isinstance(block, dict) and 'toolResult' in block - for block in content if isinstance(content, list) - ) - if not is_tool_result: - turn_start_indices.append(i) - - # With 3 turns at [0, 4, 8] and protected_turns=2, protect from index 4 onwards - turns_to_protect = min(protected_turns, len(turn_start_indices)) - protected_start_idx = turn_start_indices[-turns_to_protect] # Index 4 - - protected_indices = set(range(protected_start_idx, len(mock_messages))) - assert protected_indices == {4, 5, 6, 7, 8, 9} - def test_checkpoint_calculation(self, mock_messages): """Test checkpoint is set at oldest protected turn boundary""" protected_turns = 2 diff --git a/backend/src/agents/main_agent/session/tests/test_compaction_integration.py b/backend/src/agents/main_agent/session/tests/test_compaction_integration.py index 1bb2a93e9..de17cb9a8 100644 --- a/backend/src/agents/main_agent/session/tests/test_compaction_integration.py +++ b/backend/src/agents/main_agent/session/tests/test_compaction_integration.py @@ -179,13 +179,6 @@ def test_compaction_with_messages(): logger.info(f"Added {len(test_messages)} test messages") logger.info(f"Total messages: {len(agent.messages)}") - # Test truncation - protected_indices = session_manager._find_protected_indices( - agent.messages, - session_manager.compaction_config.protected_turns - ) - logger.info(f"Protected indices: {protected_indices}") - # Test valid cutoff indices valid_indices = session_manager._find_valid_cutoff_indices(agent.messages) logger.info(f"Valid cutoff indices: {valid_indices}") diff --git a/backend/src/agents/main_agent/session/turn_based_session_manager.py b/backend/src/agents/main_agent/session/turn_based_session_manager.py index e1c98f799..8293b6019 100644 --- a/backend/src/agents/main_agent/session/turn_based_session_manager.py +++ b/backend/src/agents/main_agent/session/turn_based_session_manager.py @@ -7,9 +7,18 @@ after the SDK finishes its standard session restore. Compaction Strategy (two-feature approach): -- Stage 1: Tool content truncation — applied every turn, reduces verbose tool I/O +- Stage 1: Tool content truncation — applied only below the persisted + truncation anchor, which moves at checkpoint advances or when the Bedrock + prompt cache has already expired between turns - Stage 2: Checkpoint + Summary — triggered when token threshold exceeded +Byte-stability contract: between compaction-state changes, restoring the same +stored history must produce byte-identical ``agent.messages``. Bedrock prompt +caching requires an exact prefix match, so any per-restore mutation of older +turns (e.g. a sliding truncation window) breaks the cached prefix and forces a +full re-write (~$2.5/MTok on a 35k–150k prefix) nearly every turn — far more +expensive than the read tokens truncation saves. + Based on: https://github.com/aws-samples/sample-strands-agent-with-agentcore """ @@ -43,7 +52,8 @@ class TurnBasedSessionManager(AgentCoreMemorySessionManager): Features: - Checkpoint-based message loading (skip old messages, prepend summary) - - Tool content truncation (reduce verbose tool I/O in older turns) + - Tool content truncation below a persisted anchor (cache-safe: history + is byte-stable between compaction-state changes) - Session cancellation support (via cancelled flag) - Compaction state persisted in DynamoDB session metadata """ @@ -235,6 +245,9 @@ def initialize(self, agent: "Agent", **kwargs: Any) -> None: except Exception as e: logger.error(f"Compaction failed, using full history: {e}", exc_info=True) self.compaction_state = CompactionState() + # Force update_after_turn to re-load persisted state rather than + # saving these defaults over the real checkpoint/anchor. + self._compaction_state_loaded = False self._valid_cutoff_indices = [] self._all_messages_for_summary = [] @@ -257,7 +270,16 @@ def _apply_compaction(self, agent: "Agent") -> None: Modifies agent.messages in-place to: 1. Skip old messages (checkpoint-based) 2. Prepend conversation summary - 3. Truncate verbose tool content in older turns + 3. Truncate tool content strictly below the persisted truncation anchor + + Byte-stability contract: this derivation must be a pure function of + (stored history, persisted compaction state). Truncation is therefore + driven ONLY by ``compaction_state.truncation_anchor`` — never by a + window computed from the current message count, which would re-mutate + an older turn on every restore and break Bedrock's exact-prefix cache + match. The anchor moves at checkpoint advances (``update_after_turn``, + where the slice already pays the one cache re-write) and + opportunistically here when the prompt cache has already expired. """ all_messages = agent.messages @@ -278,12 +300,20 @@ def _apply_compaction(self, agent: "Agent") -> None: # if update_after_turn actually advances the checkpoint) self._all_messages_for_summary = all_messages[:] + # Advance the truncation anchor only while the prompt cache is + # already cold — the prefix re-write is unavoidable then, so pending + # truncations are free. Persisted before use so subsequent restores + # derive the identical history. + self._maybe_advance_truncation_anchor() + # Apply checkpoint: skip old messages, prepend summary checkpoint = self.compaction_state.checkpoint stage = "none" + offset = 0 if checkpoint > 0 and checkpoint < len(all_messages): messages_to_process = all_messages[checkpoint:] + offset = checkpoint summary = self.compaction_state.summary if summary and messages_to_process: @@ -294,26 +324,79 @@ def _apply_compaction(self, agent: "Agent") -> None: else: messages_to_process = all_messages - # Apply truncation (always when compaction enabled) - protected_indices = self._find_protected_indices( - messages_to_process, self.compaction_config.protected_turns - ) - truncated_messages, truncation_count, _ = self._truncate_tool_contents( - messages_to_process, protected_indices=protected_indices - ) - - if truncation_count > 0: - stage = "checkpoint+truncation" if stage == "checkpoint" else "truncation" + # Truncate only messages strictly below the anchor (absolute index), + # translated into post-slice coordinates via the checkpoint offset. + truncation_count = 0 + anchor = min(max(self.compaction_state.truncation_anchor, offset), len(all_messages)) + if anchor > offset: + protected_indices = set(range(anchor - offset, len(messages_to_process))) + messages_to_process, truncation_count, _ = self._truncate_tool_contents( + messages_to_process, protected_indices=protected_indices + ) + if truncation_count > 0: + stage = "checkpoint+truncation" if stage == "checkpoint" else "truncation" - agent.messages = truncated_messages + agent.messages = messages_to_process logger.info( f"Compaction initialized: stage={stage}, " f"original={self._total_message_count_at_init}, " f"final={len(agent.messages)}, " + f"anchor={self.compaction_state.truncation_anchor}, " f"truncations={truncation_count}" ) + def _maybe_advance_truncation_anchor(self) -> None: + """Advance the truncation anchor while the prompt cache is already cold. + + The anchor normally moves only when the checkpoint advances (that + slice already forces one full prefix re-write, so folding truncation + into it costs nothing extra). But when more than + ``cache_ttl_seconds`` have passed since the previous turn, the + Bedrock prompt-cache entry has expired anyway — the next call + re-writes the prefix regardless — so the anchor can slide up to the + protected-turns boundary for free. Persisted immediately so every + subsequent restore derives the identical truncated history. + """ + state = self.compaction_state + config = self.compaction_config + if not self._cache_window_expired(state.updated_at, config.cache_ttl_seconds): + return + + cutoffs = self._valid_cutoff_indices + if len(cutoffs) <= config.protected_turns: + return + + sliding_anchor = cutoffs[-config.protected_turns] + if sliding_anchor <= max(state.truncation_anchor, state.checkpoint): + return + + logger.info( + "Truncation anchor advance (cache expired): %d -> %d", + state.truncation_anchor, sliding_anchor, + ) + state.truncation_anchor = sliding_anchor + self._save_compaction_state(state) + + @staticmethod + def _cache_window_expired(updated_at: Optional[str], ttl_seconds: int) -> bool: + """True when the previous turn is older than the prompt-cache TTL. + + ``updated_at`` is stamped by ``_save_compaction_state`` on every turn, + so it is a faithful proxy for the previous model call. Unparseable or + missing timestamps return False (conservative: assume the cache may + still be warm and leave history untouched). + """ + if not updated_at: + return False + try: + last = datetime.fromisoformat(updated_at.replace("Z", "+00:00")) + except (ValueError, TypeError): + return False + if last.tzinfo is None: + last = last.replace(tzinfo=timezone.utc) + return (datetime.now(timezone.utc) - last).total_seconds() > ttl_seconds + # ========================================================================= # Compaction State Persistence # ========================================================================= @@ -642,6 +725,12 @@ async def update_after_turn( summary = self._generate_fallback_summary(messages_to_summarize) self.compaction_state.checkpoint = new_checkpoint + # The anchor rides the checkpoint: everything the slice retains stays + # byte-identical until the next compaction-state change, so the single + # mutation (slice + summary) is paid with exactly one cache re-write. + self.compaction_state.truncation_anchor = max( + self.compaction_state.truncation_anchor, new_checkpoint + ) self.compaction_state.summary = summary # Running total persisted alongside the rest of the compaction state # so a refresh can rehydrate the end-of-conversation summary indicator. @@ -737,19 +826,6 @@ def _find_valid_cutoff_indices(self, messages: List[Dict]) -> List[int]: valid_indices.append(i) return valid_indices - def _find_protected_indices(self, messages: List[Dict], protected_turns: int) -> set: - """Find message indices that should be protected from truncation.""" - if protected_turns <= 0: - return set() - - turn_start_indices = self._find_valid_cutoff_indices(messages) - if not turn_start_indices: - return set() - - turns_to_protect = min(protected_turns, len(turn_start_indices)) - protected_start_idx = turn_start_indices[-turns_to_protect] - return set(range(protected_start_idx, len(messages))) - # ========================================================================= # Truncation (Stage 1 Compaction) # ========================================================================= diff --git a/backend/src/agents/main_agent/skills/strands_mapping.py b/backend/src/agents/main_agent/skills/strands_mapping.py index 2f04c4cad..2b395fe71 100644 --- a/backend/src/agents/main_agent/skills/strands_mapping.py +++ b/backend/src/agents/main_agent/skills/strands_mapping.py @@ -262,6 +262,11 @@ def build_skills_runtime( if not records: return None, None + # Deterministic order regardless of fetch order: these records become the + # system-prompt block, and any order flip between turns + # of a session invalidates the Bedrock prompt cache (exact-prefix match). + records = sorted(records, key=lambda r: getattr(r, "skill_id", "") or "") + skills = [record_to_strands_skill(r) for r in records] plugin = AgentSkills(skills=skills) read_tool = make_read_skill_file_tool(records) diff --git a/backend/src/agents/main_agent/streaming/stream_coordinator.py b/backend/src/agents/main_agent/streaming/stream_coordinator.py index 4f4174f9b..0b4731935 100644 --- a/backend/src/agents/main_agent/streaming/stream_coordinator.py +++ b/backend/src/agents/main_agent/streaming/stream_coordinator.py @@ -11,6 +11,10 @@ from typing import Any, AsyncGenerator, Dict, List, Optional, Union from agents.main_agent.config.constants import EnvVars +from agents.main_agent.session.hooks.prefix_fingerprint import ( + get_prefix_fingerprint, + reset_prefix_fingerprints, +) from apis.shared.errors import ( ConversationalErrorEvent, ErrorCode, @@ -81,6 +85,11 @@ async def stream_response( os.environ[EnvVars.SESSION_ID] = session_id os.environ[EnvVars.USER_ID] = user_id + # Per-turn prompt-cache prefix fingerprints: clear the previous + # turn's entries so entry N of this turn maps to the turn's Nth + # model call (the agent instance is cached across turns). + reset_prefix_fingerprints(agent) + # Track timing for latency metrics stream_start_time = time.time() # Wall-clock turn start as a tz-aware datetime. Used post-turn to @@ -933,19 +942,24 @@ async def stream_response( first_token_time=first_token_for_message, agent=main_agent_wrapper, # Use wrapper instead of internal agent citations=citations_for_message, # Pass citations for persistence + call_index=idx, # Nth model call of this turn (prefix fingerprint lookup) ) ) - # Execute all metadata storage tasks in parallel - # Use return_exceptions=True to prevent one failure from cancelling others - if metadata_tasks: - results = await asyncio.gather(*metadata_tasks, return_exceptions=True) - # Log any failures (but don't raise - metadata failures shouldn't break streaming) - for idx, result in enumerate(results): - if isinstance(result, Exception): - logger.error(f"Failed to store metadata for message {message_ids_to_store[idx]}: {result}") + # Execute metadata storage tasks SEQUENTIALLY, in call order. + # The write path derives each call's cacheStatus from the + # session's previous cost row; parallel writes would race the + # turn's own rows and misclassify calls 2..N. These are + # post-stream background writes, so the latency cost is + # invisible to the user. + for idx, task in enumerate(metadata_tasks): + try: + await task + except Exception as task_error: + # Log but don't raise - metadata failures shouldn't break streaming + logger.error(f"Failed to store metadata for message {message_ids_to_store[idx]}: {task_error}") - logger.info(f"✅ Message metadata stored for {len(message_ids_to_store)} assistant messages (parallel)") + logger.info(f"✅ Message metadata stored for {len(message_ids_to_store)} assistant messages (sequential)") # Store displayText for user message if original_message differs from augmented if original_message: @@ -2072,6 +2086,7 @@ async def _store_message_metadata( first_token_time: Optional[float], agent: Any = None, citations: Optional[List] = None, + call_index: Optional[int] = None, ) -> None: """ Store message-level metadata (token usage, latency, model info, citations) @@ -2086,6 +2101,10 @@ async def _store_message_metadata( first_token_time: Timestamp of first token received agent: Agent instance for extracting model info citations: Optional list of citation dicts from RAG retrieval + call_index: Index of this model call within the turn, used to + look up the matching prompt-cache prefix fingerprint stashed + by PrefixFingerprintHook. None → latest fingerprint (single- + call paths like interrupted-turn persistence). """ try: from apis.shared.sessions.models import Attribution, LatencyMetrics, MessageMetadata, ModelInfo, TokenUsage @@ -2219,6 +2238,16 @@ async def _store_message_metadata( ) if context_window is not None: metadata_kwargs["contextWindow"] = context_window + + # Prompt-cache prefix fingerprints for this model call + # (extra field via extra="allow"; persisted on the cost row + # so cache misses are diagnosable component-by-component). + strands_agent = getattr(agent, "agent", None) if agent is not None else None + if strands_agent is not None: + prefix_fingerprint = get_prefix_fingerprint(strands_agent, call_index) + if prefix_fingerprint: + metadata_kwargs["prefixFingerprints"] = prefix_fingerprint + message_metadata = MessageMetadata(**metadata_kwargs) # Store metadata diff --git a/backend/src/apis/app_api/admin/costs/models.py b/backend/src/apis/app_api/admin/costs/models.py index 6791b4596..b8c7cbaa7 100644 --- a/backend/src/apis/app_api/admin/costs/models.py +++ b/backend/src/apis/app_api/admin/costs/models.py @@ -96,6 +96,55 @@ class CostTrend(BaseModel): active_users: int = Field(..., alias="activeUsers") +class PrefixFingerprints(BaseModel): + """Prompt-cache prefix hashes for one model call (see PrefixFingerprintHook).""" + model_config = ConfigDict(populate_by_name=True) + + tool_config_hash: Optional[str] = Field(None, alias="toolConfigHash") + system_prompt_hash: Optional[str] = Field(None, alias="systemPromptHash") + history_hash: Optional[str] = Field(None, alias="historyHash") + message_count: Optional[int] = Field(None, alias="messageCount") + + +class SessionCallRow(BaseModel): + """One model call within a session's cost anatomy.""" + model_config = ConfigDict(populate_by_name=True) + + timestamp: str + message_id: Optional[int] = Field(None, alias="messageId") + model_id: Optional[str] = Field(None, alias="modelId") + + input_tokens: int = Field(0, alias="inputTokens") + output_tokens: int = Field(0, alias="outputTokens") + cache_read_tokens: int = Field(0, alias="cacheReadTokens") + cache_write_tokens: int = Field(0, alias="cacheWriteTokens") + + cost: float = 0.0 + cache_status: Optional[str] = Field(None, alias="cacheStatus") + cache_gap_seconds: Optional[int] = Field(None, alias="cacheGapSeconds") + wasted_usd: float = Field(0.0, alias="wastedUsd") + prefix_fingerprints: Optional[PrefixFingerprints] = Field( + None, alias="prefixFingerprints" + ) + + +class SessionCostAnatomy(BaseModel): + """Per-call cost anatomy for one session (admin cache-miss forensics).""" + model_config = ConfigDict(populate_by_name=True) + + session_id: str = Field(..., alias="sessionId") + calls: List[SessionCallRow] + + total_cost: float = Field(0.0, alias="totalCost") + total_cache_read_tokens: int = Field(0, alias="totalCacheReadTokens") + total_cache_write_tokens: int = Field(0, alias="totalCacheWriteTokens") + avoidable_miss_count: int = Field(0, alias="avoidableMissCount") + wasted_usd: float = Field(0.0, alias="wastedUsd") + # cacheRead / (cacheRead + cacheWrite) over the session; None until + # there has been any cache activity. + cache_efficiency: Optional[float] = Field(None, alias="cacheEfficiency") + + class AdminCostDashboard(BaseModel): """Complete admin cost dashboard response combining all metrics.""" model_config = ConfigDict(populate_by_name=True) diff --git a/backend/src/apis/app_api/admin/costs/routes.py b/backend/src/apis/app_api/admin/costs/routes.py index 8c451751f..5d8600432 100644 --- a/backend/src/apis/app_api/admin/costs/routes.py +++ b/backend/src/apis/app_api/admin/costs/routes.py @@ -22,6 +22,7 @@ TierUsageSummary, CostTrend, AdminCostDashboard, + SessionCostAnatomy, ) from .service import AdminCostService @@ -109,6 +110,60 @@ async def get_cost_dashboard( ) +@router.get("/sessions/{session_id}/calls", response_model=SessionCostAnatomy) +async def get_session_cost_anatomy( + session_id: str, + admin_user: User = Depends(require_admin), + service: AdminCostService = Depends(get_cost_service) +): + """ + Get the per-model-call "cost anatomy" of a session. + + Returns one row per model call — timestamp, input/cacheRead/cacheWrite/ + output tokens, cost, derived cacheStatus (first_write | hit | + miss_ttl_expired | miss_avoidable | uncached), and the prompt-cache + prefix fingerprint hashes (toolConfig / system prompt / history) — plus + session-level rollups (cache efficiency, avoidable-miss count, wastedUsd). + + This is the forensic view for diagnosing avoidable prompt-cache + re-writes: on a miss_avoidable row, diff its fingerprint hashes against + the previous row's to see which prefix component changed. + + Args: + session_id: Session identifier (any user's session — admin scope) + admin_user: Authenticated admin user (injected) + service: Admin cost service (injected) + + Returns: + SessionCostAnatomy with chronological per-call rows + + Raises: + HTTPException: + - 401 if not authenticated + - 403 if user lacks admin role + - 404 if the session has no cost records + - 500 if server error + """ + logger.info("Admin requesting session cost anatomy") + + try: + anatomy = await service.get_session_cost_anatomy(session_id) + except Exception as e: + logger.error(f"Error getting session cost anatomy: {e}", exc_info=True) + raise HTTPException( + status_code=500, + detail="Failed to retrieve session cost anatomy" + ) + + if not anatomy.calls: + raise HTTPException( + status_code=404, + detail=f"No cost records found for session {session_id}" + ) + + return anatomy + + @router.get("/top-users", response_model=List[TopUserCost]) async def get_top_users( period: Optional[str] = Query( diff --git a/backend/src/apis/app_api/admin/costs/service.py b/backend/src/apis/app_api/admin/costs/service.py index 08d67922b..def19ba56 100644 --- a/backend/src/apis/app_api/admin/costs/service.py +++ b/backend/src/apis/app_api/admin/costs/service.py @@ -16,6 +16,9 @@ TierUsageSummary, CostTrend, AdminCostDashboard, + PrefixFingerprints, + SessionCallRow, + SessionCostAnatomy, ) logger = logging.getLogger(__name__) @@ -313,6 +316,99 @@ async def get_daily_trends( logger.error(f"Error getting daily trends: {e}") raise + async def get_session_cost_anatomy(self, session_id: str) -> SessionCostAnatomy: + """ + Get the per-model-call cost anatomy for one session. + + Reads every C# cost record for the session (chronological) and maps + each to a SessionCallRow with token splits, cost, derived cacheStatus, + and the prompt-cache prefix fingerprints — the data needed to see + where a session's spend went and which prefix component broke the + cache on a miss. Rows written before this feature shipped simply lack + cacheStatus/fingerprints and render as nulls. + + Args: + session_id: Session identifier (any user's — admin scope). + + Returns: + SessionCostAnatomy with per-call rows and session-level rollups. + """ + records = await self.storage.get_session_cost_records(session_id) + + calls: List[SessionCallRow] = [] + total_cost = 0.0 + total_cache_read = 0 + total_cache_write = 0 + avoidable_misses = 0 + wasted_usd = 0.0 + + for record in records: + token_usage = record.get("tokenUsage") or {} + model_info = record.get("modelInfo") or {} + fingerprints_raw = record.get("prefixFingerprints") + + # cost is a breakdown dict ({"total": ...}) on the streaming path + # or a bare float on the legacy path. + cost_raw = record.get("cost") + if isinstance(cost_raw, dict): + cost_raw = cost_raw.get("total") + try: + cost = float(cost_raw) if cost_raw is not None else 0.0 + except (TypeError, ValueError): + cost = 0.0 + + cache_read = int(token_usage.get("cacheReadInputTokens") or 0) + cache_write = int(token_usage.get("cacheWriteInputTokens") or 0) + cache_status = record.get("cacheStatus") + row_wasted = float(record.get("wastedUsd") or 0.0) + + total_cost += cost + total_cache_read += cache_read + total_cache_write += cache_write + if cache_status == "miss_avoidable": + avoidable_misses += 1 + wasted_usd += row_wasted + + gap_raw = record.get("cacheGapSeconds") + calls.append(SessionCallRow( + timestamp=record.get("timestamp", ""), + message_id=record.get("messageId"), + model_id=model_info.get("modelId"), + input_tokens=int(token_usage.get("inputTokens") or 0), + output_tokens=int(token_usage.get("outputTokens") or 0), + cache_read_tokens=cache_read, + cache_write_tokens=cache_write, + cost=cost, + cache_status=cache_status, + cache_gap_seconds=int(gap_raw) if gap_raw is not None else None, + wasted_usd=row_wasted, + prefix_fingerprints=( + PrefixFingerprints(**fingerprints_raw) + if isinstance(fingerprints_raw, dict) else None + ), + )) + + cache_traffic = total_cache_read + total_cache_write + cache_efficiency = ( + total_cache_read / cache_traffic if cache_traffic > 0 else None + ) + + logger.info( + f"Session cost anatomy: {len(calls)} calls, " + f"{avoidable_misses} avoidable misses, wasted=${wasted_usd:.4f}" + ) + + return SessionCostAnatomy( + session_id=session_id, + calls=calls, + total_cost=round(total_cost, 6), + total_cache_read_tokens=total_cache_read, + total_cache_write_tokens=total_cache_write, + avoidable_miss_count=avoidable_misses, + wasted_usd=round(wasted_usd, 6), + cache_efficiency=cache_efficiency, + ) + async def get_dashboard( self, period: Optional[str] = None, diff --git a/backend/src/apis/shared/observability/__init__.py b/backend/src/apis/shared/observability/__init__.py new file mode 100644 index 000000000..1acf860c6 --- /dev/null +++ b/backend/src/apis/shared/observability/__init__.py @@ -0,0 +1,25 @@ +"""Shared observability helpers (prompt-cache economics, EMF metrics).""" + +from apis.shared.observability.prompt_cache import ( + CACHE_TTL_SECONDS, + PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, + prompt_cache_observability_enabled, + CacheStatus, + classify_cache_status, + compute_wasted_usd, + fingerprint_canonical_json, + fingerprint_text, +) +from apis.shared.observability.emf import emit_prompt_cache_metrics + +__all__ = [ + "CACHE_TTL_SECONDS", + "PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV", + "prompt_cache_observability_enabled", + "CacheStatus", + "classify_cache_status", + "compute_wasted_usd", + "fingerprint_canonical_json", + "fingerprint_text", + "emit_prompt_cache_metrics", +] diff --git a/backend/src/apis/shared/observability/emf.py b/backend/src/apis/shared/observability/emf.py new file mode 100644 index 000000000..ab17944bf --- /dev/null +++ b/backend/src/apis/shared/observability/emf.py @@ -0,0 +1,89 @@ +"""CloudWatch Embedded Metric Format (EMF) emission for prompt-cache metrics. + +EMF turns a structured log line into CloudWatch metrics with no SDK calls, +no batching agent, and no extra IAM: CloudWatch Logs extracts any log event +whose message is a JSON object carrying the ``_aws.CloudWatchMetrics`` +directive. Both compute surfaces already ship stdout to CloudWatch Logs +(inference-api via its AgentCore Runtime log group, app-api via ECS awslogs), +so a raw JSON line on stdout is all that's needed. + +The line must be *exactly* the JSON object — a ``[INFO] logger-name:`` prefix +from the app's standard formatter would break extraction — so this module +uses a dedicated non-propagating logger with a message-only formatter. +""" + +import json +import logging +import os +import sys +import time +from typing import Optional + +_EMF_NAMESPACE = os.environ.get("EMF_NAMESPACE", "AgentCoreStack/PromptCache") + +# Dedicated raw-JSON stdout logger. propagate=False keeps the app-level +# formatter (and its non-JSON prefixes) away from these lines. +_emf_logger = logging.getLogger("apis.shared.observability.emf.raw") +if not _emf_logger.handlers: + _handler = logging.StreamHandler(sys.stdout) + _handler.setFormatter(logging.Formatter("%(message)s")) + _emf_logger.addHandler(_handler) + _emf_logger.setLevel(logging.INFO) + _emf_logger.propagate = False + +logger = logging.getLogger(__name__) + + +def emit_prompt_cache_metrics( + cache_read_tokens: int, + cache_write_tokens: int, + avoidable_miss: bool, + wasted_usd: float = 0.0, + model_id: Optional[str] = None, + session_id: Optional[str] = None, + cache_status: Optional[str] = None, +) -> None: + """Emit one EMF record for a completed model call. + + Metrics (no dimensions — fleet-wide sums are the alerting target; the + per-model/per-session detail rides along as queryable log properties): + + - ``CacheReadTokens`` / ``CacheWriteTokens``: fleet cache traffic; their + ratio is the cache-efficiency dashboard line. + - ``AvoidableMiss``: count of calls classified ``miss_avoidable`` — the + alarm target (a prefix-stability regression shows up as a step change). + - ``WastedUsd``: dollars attributed to avoidable re-writes. + + Best-effort: never raises. + """ + try: + record = { + "_aws": { + "Timestamp": int(time.time() * 1000), + "CloudWatchMetrics": [ + { + "Namespace": _EMF_NAMESPACE, + "Dimensions": [[]], + "Metrics": [ + {"Name": "CacheReadTokens", "Unit": "Count"}, + {"Name": "CacheWriteTokens", "Unit": "Count"}, + {"Name": "AvoidableMiss", "Unit": "Count"}, + {"Name": "WastedUsd", "Unit": "None"}, + ], + } + ], + }, + "CacheReadTokens": int(cache_read_tokens or 0), + "CacheWriteTokens": int(cache_write_tokens or 0), + "AvoidableMiss": 1 if avoidable_miss else 0, + "WastedUsd": round(float(wasted_usd or 0.0), 6), + } + if model_id: + record["modelId"] = model_id + if session_id: + record["sessionId"] = session_id + if cache_status: + record["cacheStatus"] = cache_status + _emf_logger.info(json.dumps(record, separators=(",", ":"))) + except Exception as e: # noqa: BLE001 - metrics must never break a request + logger.debug("EMF emission skipped: %s", e) diff --git a/backend/src/apis/shared/observability/prompt_cache.py b/backend/src/apis/shared/observability/prompt_cache.py new file mode 100644 index 000000000..712ddfbfb --- /dev/null +++ b/backend/src/apis/shared/observability/prompt_cache.py @@ -0,0 +1,169 @@ +"""Prompt-cache economics: prefix fingerprints and per-call cache classification. + +Bedrock prompt caching is exact-prefix-match with a ~5-minute sliding TTL. +When the request prefix (toolConfig + system prompt + prior message history) +is byte-identical to the previous call's, tokens are read from cache at a +steep discount; any divergence forces a full cache re-write at a premium. +A production audit (session aecd387d, $1.60) showed 75% of spend was +avoidable re-writes caused by nondeterministic prefix assembly — and proving +that took hours of manual forensics. This module makes the whole class +measurable: + +- ``fingerprint_*`` produce short stable hashes of the three prefix + components so consecutive metadata rows show *which* component changed + when a cache miss happens. +- ``classify_cache_status`` labels each model call from its token usage and + the previous call's row. +- ``compute_wasted_usd`` prices the avoidable portion of a re-write. + +Pure functions only — no AWS calls — so they are unit-testable and safe to +import from any package (agents/, app_api, inference_api all may import +``apis.shared``). +""" + +import hashlib +import json +import os +from enum import Enum +from typing import Any, Mapping, Optional + +# Kill switch for the whole prompt-cache observability layer (fingerprint +# hook, per-call cacheStatus derivation + session rollups, EMF emission). +# Default ON; only the literal string "false" disables it — an empty or +# unset value stays enabled (workflow env vars can materialize as ""). +PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV = "PROMPT_CACHE_OBSERVABILITY_ENABLED" + + +def prompt_cache_observability_enabled() -> bool: + """Whether prompt-cache observability is enabled (env kill switch). + + Read per call (no module-level caching) so tests and live config changes + behave predictably; the env read is negligible next to the DynamoDB + lookup and hash work it gates. + """ + return os.environ.get(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, "").lower() != "false" + +# Bedrock prompt-cache TTL (sliding, seconds). A gap between consecutive +# model calls larger than this means the cache entry legitimately expired — +# the re-write was unavoidable. +CACHE_TTL_SECONDS = 300 + + +class CacheStatus(str, Enum): + """Derived per-call prompt-cache outcome. + + - ``first_write``: the initial, expected cache population — either no + previous call row for the session, or every activity-bearing signal + says no cache existed yet (the previous call was ``uncached``, e.g. + the prompt was below the model's minimum cacheable prefix, so there + was nothing to read from). + - ``hit``: tokens were read from cache (``cacheReadInputTokens > 0``). + Partial re-writes of a changed suffix still count as hits. + - ``miss_ttl_expired``: nothing read, cache re-written, and the gap since + the previous call exceeded the cache TTL — unavoidable. + - ``miss_avoidable``: nothing read, cache re-written, previous call had a + live cache entry within the TTL — the prefix must have changed. This is + the bug class the fingerprints exist to diagnose. + - ``uncached``: no cache activity at all (caching disabled, non-Bedrock + provider, or prompt below the minimum cacheable length). + """ + + FIRST_WRITE = "first_write" + HIT = "hit" + MISS_TTL_EXPIRED = "miss_ttl_expired" + MISS_AVOIDABLE = "miss_avoidable" + UNCACHED = "uncached" + + +def fingerprint_text(text: Optional[str]) -> str: + """Short stable hash of a text blob (e.g. the system prompt).""" + return hashlib.sha256((text or "").encode("utf-8")).hexdigest()[:16] + + +def fingerprint_canonical_json(obj: Any) -> str: + """Short stable hash of a JSON-serializable structure. + + Dict keys are sorted so key insertion order never changes the hash, but + list order is preserved — deliberately, because list order (tool specs, + message history, content blocks) is exactly what Bedrock's exact-prefix + match is sensitive to. + """ + canonical = json.dumps(obj, sort_keys=True, separators=(",", ":"), default=str) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16] + + +def classify_cache_status( + cache_read_tokens: int, + cache_write_tokens: int, + previous_call_exists: bool, + gap_seconds: Optional[float], + previous_cached_prefix_tokens: Optional[int] = None, +) -> CacheStatus: + """Classify one model call's cache outcome. + + Args: + cache_read_tokens: ``cacheReadInputTokens`` for this call. + cache_write_tokens: ``cacheWriteInputTokens`` for this call. + previous_call_exists: Whether the session has an earlier call row. + gap_seconds: Seconds since the previous call row's timestamp; None + when unknown (treated conservatively as expired, not avoidable). + previous_cached_prefix_tokens: Previous call's cacheRead + cacheWrite + token total, or None when unknown. Zero means the previous call + was uncached (e.g. prompt below the model's minimum cacheable + prefix), so no cache entry existed for this call to read. + """ + if cache_read_tokens > 0: + return CacheStatus.HIT + if cache_write_tokens <= 0: + return CacheStatus.UNCACHED + if not previous_call_exists: + return CacheStatus.FIRST_WRITE + if previous_cached_prefix_tokens is not None and previous_cached_prefix_tokens <= 0: + # The previous call wrote nothing to the cache, so there was no entry + # to read from — this write is the session's first real population + # (typically the first prompt to cross the minimum cacheable length), + # not a miss of any kind. + return CacheStatus.FIRST_WRITE + if gap_seconds is None or gap_seconds > CACHE_TTL_SECONDS: + return CacheStatus.MISS_TTL_EXPIRED + return CacheStatus.MISS_AVOIDABLE + + +def compute_wasted_usd( + cache_status: CacheStatus, + cache_write_tokens: int, + previous_cached_prefix_tokens: Optional[int], + pricing_snapshot: Optional[Mapping[str, Any]], +) -> float: + """USD wasted by an avoidable cache re-write; 0.0 for every other status. + + The waste is the re-written portion of the prefix that was already cached + on the previous call (``min(cacheWrite, previous cacheRead + cacheWrite)``; + falls back to the full cacheWrite when the previous split is unknown), + priced at the cache-write premium over the cache-read rate the tokens + *should* have cost. + + Args: + cache_status: Result of :func:`classify_cache_status`. + cache_write_tokens: ``cacheWriteInputTokens`` for this call. + previous_cached_prefix_tokens: Previous call's cacheRead + cacheWrite + token total, or None when unavailable. + pricing_snapshot: The row's ``pricingSnapshot`` dict (camelCase keys). + """ + if cache_status is not CacheStatus.MISS_AVOIDABLE: + return 0.0 + if not pricing_snapshot or cache_write_tokens <= 0: + return 0.0 + + # `or 0` (not `.get(..., 0)`) — rows can store an explicit None. + write_price = pricing_snapshot.get("cacheWritePricePerMtok") or 0 + read_price = pricing_snapshot.get("cacheReadPricePerMtok") or 0 + premium_per_mtok = write_price - read_price + if premium_per_mtok <= 0: + return 0.0 + + rewritten = cache_write_tokens + if previous_cached_prefix_tokens is not None and previous_cached_prefix_tokens > 0: + rewritten = min(cache_write_tokens, previous_cached_prefix_tokens) + + return (rewritten / 1_000_000) * premium_per_mtok diff --git a/backend/src/apis/shared/rbac/service.py b/backend/src/apis/shared/rbac/service.py index 4c1f93e80..8b937c22f 100644 --- a/backend/src/apis/shared/rbac/service.py +++ b/backend/src/apis/shared/rbac/service.py @@ -192,9 +192,13 @@ def _merge_permissions( return UserEffectivePermissions( user_id=user_id, app_roles=[r.role_id for r in roles], - tools=list(all_tools), - models=list(all_models), - skills=list(all_skills), + # Sorted for a deterministic order across processes (set iteration + # varies with hash randomization). These lists reach the model's + # system prompt / tool config, where an order flip between turns + # invalidates the Bedrock prompt cache. + tools=sorted(all_tools), + models=sorted(all_models), + skills=sorted(all_skills), quota_tier=quota_tier, resolved_at=datetime.now(timezone.utc).isoformat() + "Z", ) diff --git a/backend/src/apis/shared/sessions/metadata.py b/backend/src/apis/shared/sessions/metadata.py index bc75d0c9e..7fc1f5225 100644 --- a/backend/src/apis/shared/sessions/metadata.py +++ b/backend/src/apis/shared/sessions/metadata.py @@ -237,6 +237,16 @@ async def _store_message_metadata_cloud( # Extract timestamp for SK and GSI timestamp = metadata_dict.get("attribution", {}).get("timestamp", datetime.now(timezone.utc).isoformat()) + # Derive prompt-cache observability (cacheStatus / wastedUsd) from + # this call's usage + the session's previous cost row. Best-effort: + # {} on any failure so it can never block the critical write. + cache_observability = _derive_cache_observability( + session_id=session_id, + table=table, + timestamp=timestamp, + message_metadata=message_metadata, + ) + # Generate unique ID for SK to prevent collisions unique_id = str(uuid_lib.uuid4()) @@ -270,7 +280,11 @@ async def _store_message_metadata_cloud( "ttl": ttl, # Cost and usage metadata - **metadata_decimal + **metadata_decimal, + + # Derived prompt-cache observability (cacheStatus, cacheGapSeconds, + # wastedUsd) — {} when derivation was skipped or failed + **_convert_floats_to_decimal(cache_observability), } # Store in DynamoDB @@ -279,14 +293,21 @@ async def _store_message_metadata_cloud( logger.info(f"💾 Stored cost record in DynamoDB table {table_name}") logger.info(f" Session: {session_id}, Message: {message_id}, SK: C#{timestamp}#{unique_id[:8]}...") + # Per-call EMF metrics (CacheReadTokens / CacheWriteTokens / + # AvoidableMiss / WastedUsd) for the fleet cache-efficiency + # dashboard + alarm. Best-effort, never raises. + _emit_cache_metrics(session_id, message_metadata, cache_observability) + # Bump session-level aggregates (totalCost, lastContextTokens, - # contextWindow) for the session-cost badge. Best-effort — drift is - # repaired by lazy backfill on the next metadata read. + # contextWindow, cache-efficiency counters) for the session-cost + # badge and admin lists. Best-effort — drift is repaired by lazy + # backfill on the next metadata read. await _bump_session_aggregates( session_id=session_id, user_id=user_id, message_metadata=message_metadata, table=table, + cache_observability=cache_observability, ) # Update pre-aggregated cost summary for fast quota checks @@ -313,6 +334,174 @@ async def _store_message_metadata_cloud( +def _extract_cache_usage(message_metadata: MessageMetadata) -> Tuple[int, int]: + """Return (cache_read_tokens, cache_write_tokens) from a metadata object.""" + token_usage = message_metadata.token_usage + if not token_usage: + return 0, 0 + return ( + token_usage.cache_read_input_tokens or 0, + token_usage.cache_write_input_tokens or 0, + ) + + +def _extract_pricing_dict(message_metadata: MessageMetadata) -> Optional[Dict[str, Any]]: + """Return the pricingSnapshot as a camelCase dict, or None.""" + if not message_metadata.model_info: + return None + pricing = message_metadata.model_info.pricing_snapshot + if pricing is None: + return None + if hasattr(pricing, "model_dump"): + return pricing.model_dump(by_alias=True) + return pricing + + +def _derive_cache_observability( + session_id: str, + table, + timestamp: str, + message_metadata: MessageMetadata, +) -> Dict[str, Any]: + """Classify this model call's prompt-cache outcome against the previous call. + + Queries the session's most recent existing ``C#`` cost row (one GSI read, + Limit=1) and derives: + + - ``cacheStatus``: first_write | hit | miss_ttl_expired | miss_avoidable + | uncached (see ``apis.shared.observability.CacheStatus``). + - ``cacheGapSeconds``: whole seconds since the previous call, when known. + - ``wastedUsd``: for avoidable misses, the re-written previously-cached + prefix priced at the cache-write premium over the cache-read rate, + using this row's own pricingSnapshot. + + The stream coordinator writes a turn's rows sequentially in call order, + so within a multi-call turn each call sees its predecessor. Returns {} + when there is no token usage to classify or on any failure — derivation + must never block the critical cost-record write. + """ + try: + from boto3.dynamodb.conditions import Key + from datetime import datetime + + from apis.shared.observability import ( + CacheStatus, + classify_cache_status, + compute_wasted_usd, + prompt_cache_observability_enabled, + ) + + # Kill switch: also skips the session cache rollups and (via the + # empty dict) the EMF emission downstream. + if not prompt_cache_observability_enabled(): + return {} + + if not message_metadata.token_usage: + return {} + + cache_read, cache_write = _extract_cache_usage(message_metadata) + + # Most recent existing cost row for this session (GSI_SK = C# + # sorts chronologically; descending scan + Limit=1 = the previous call). + response = table.query( + IndexName="SessionLookupIndex", + KeyConditionExpression=( + Key("GSI_PK").eq(f"SESSION#{session_id}") + & Key("GSI_SK").begins_with("C#") + ), + ScanIndexForward=False, + Limit=1, + ) + prev_items = response.get("Items", []) + prev_row = _convert_decimal_to_float(prev_items[0]) if prev_items else None + + gap_seconds: Optional[float] = None + prev_cached_prefix: Optional[int] = None + if prev_row: + prev_ts = prev_row.get("timestamp") + try: + current_dt = datetime.fromisoformat(timestamp.replace("Z", "+00:00")) + prev_dt = datetime.fromisoformat(str(prev_ts).replace("Z", "+00:00")) + gap_seconds = (current_dt - prev_dt).total_seconds() + except (ValueError, AttributeError, TypeError): + gap_seconds = None + + prev_usage = prev_row.get("tokenUsage") or {} + prev_cached_prefix = int( + (prev_usage.get("cacheReadInputTokens") or 0) + + (prev_usage.get("cacheWriteInputTokens") or 0) + ) + + status = classify_cache_status( + cache_read_tokens=cache_read, + cache_write_tokens=cache_write, + previous_call_exists=prev_row is not None, + gap_seconds=gap_seconds, + previous_cached_prefix_tokens=prev_cached_prefix, + ) + wasted_usd = compute_wasted_usd( + cache_status=status, + cache_write_tokens=cache_write, + previous_cached_prefix_tokens=prev_cached_prefix, + pricing_snapshot=_extract_pricing_dict(message_metadata), + ) + + result: Dict[str, Any] = { + "cacheStatus": status.value, + "wastedUsd": round(wasted_usd, 6), + } + if gap_seconds is not None and gap_seconds >= 0: + result["cacheGapSeconds"] = int(gap_seconds) + + if status is CacheStatus.MISS_AVOIDABLE: + logger.warning( + "🔥 Avoidable prompt-cache miss: session=%s gap=%ss cacheWrite=%d wasted=$%.6f", + session_id, result.get("cacheGapSeconds"), cache_write, wasted_usd, + ) + return result + + except Exception as e: + # JUSTIFICATION: cache classification is derived observability; the + # authoritative usage numbers are already on the row. Never block the + # cost-record write over it. + logger.debug("Cache observability derivation skipped: %s", e) + return {} + + +def _emit_cache_metrics( + session_id: str, + message_metadata: MessageMetadata, + cache_observability: Dict[str, Any], +) -> None: + """Emit per-call EMF metrics for the fleet cache dashboard. Never raises.""" + try: + from apis.shared.observability import ( + CacheStatus, + emit_prompt_cache_metrics, + prompt_cache_observability_enabled, + ) + + if not prompt_cache_observability_enabled(): + return + + if not message_metadata.token_usage: + return + + cache_read, cache_write = _extract_cache_usage(message_metadata) + status = cache_observability.get("cacheStatus") + emit_prompt_cache_metrics( + cache_read_tokens=cache_read, + cache_write_tokens=cache_write, + avoidable_miss=status == CacheStatus.MISS_AVOIDABLE.value, + wasted_usd=cache_observability.get("wastedUsd") or 0.0, + model_id=message_metadata.model_info.model_id if message_metadata.model_info else None, + session_id=session_id, + cache_status=status, + ) + except Exception as e: # noqa: BLE001 - metrics must never break the write path + logger.debug("Cache EMF emission skipped: %s", e) + + async def _update_cost_summary_async( user_id: str, timestamp: str, @@ -1240,6 +1429,7 @@ async def _bump_session_aggregates( user_id: str, message_metadata: MessageMetadata, table, + cache_observability: Optional[Dict[str, Any]] = None, ) -> None: """Atomically update the session row's denormalized cost + context fields. @@ -1247,6 +1437,10 @@ async def _bump_session_aggregates( ``update_item`` call: - ``ADD totalCost :c`` — concurrent-safe across overlapping turns. + - ``ADD totalCacheReadTokens / totalCacheWriteTokens / avoidableMissCount + / wastedUsd`` — per-session cache-efficiency rollups so lists and + admin views can show a cache-efficiency ratio without scanning the + session's cost rows. - ``SET lastContextTokens :t, contextWindow :w`` — last-write-wins, which is the right behavior for "most recent turn." @@ -1293,7 +1487,33 @@ async def _bump_session_aggregates( update_parts_set.append("contextWindow = :w") values[":w"] = int(context_window) - update_expression = "ADD totalCost :c SET " + ", ".join(update_parts_set) + # Per-session cache-efficiency rollups (next to totalCost). Zero + # deltas are added unconditionally so the attributes exist (as 0) + # from the session's first call — simpler consumers, no sparse-field + # handling. + cache_read = token_usage.cache_read_input_tokens or 0 if token_usage else 0 + cache_write = token_usage.cache_write_input_tokens or 0 if token_usage else 0 + observability = cache_observability or {} + is_avoidable_miss = observability.get("cacheStatus") == "miss_avoidable" + wasted_usd = observability.get("wastedUsd") or 0.0 + + update_parts_add = [ + "totalCost :c", + "totalCacheReadTokens :cacheRead", + "totalCacheWriteTokens :cacheWrite", + "avoidableMissCount :avoidableMiss", + "wastedUsd :wasted", + ] + values[":cacheRead"] = int(cache_read) + values[":cacheWrite"] = int(cache_write) + values[":avoidableMiss"] = 1 if is_avoidable_miss else 0 + # wastedUsd comes from our own compute_wasted_usd (finite, rounded), + # but coerce defensively — a bad value must not break the bump. + values[":wasted"] = Decimal(str(_coerce_cost_total(wasted_usd))) + + update_expression = ( + "ADD " + ", ".join(update_parts_add) + " SET " + ", ".join(update_parts_set) + ) table.update_item( Key={"PK": f"USER#{user_id}", "SK": sk}, diff --git a/backend/src/apis/shared/skills/repository.py b/backend/src/apis/shared/skills/repository.py index 6a83bb3f7..66bb9d688 100644 --- a/backend/src/apis/shared/skills/repository.py +++ b/backend/src/apis/shared/skills/repository.py @@ -319,7 +319,11 @@ async def batch_get_skills( skill_ids: List of skill identifiers Returns: - List of SkillDefinition objects (may be shorter if some not found) + List of SkillDefinition objects (may be shorter if some not found), + sorted by skill_id. DynamoDB batch_get_item response order is + arbitrary; these records feed the system-prompt + block, and an order flip between turns invalidates the Bedrock + prompt cache (exact-prefix match). """ if not skill_ids: return [] @@ -342,7 +346,7 @@ async def batch_get_skills( [SkillDefinition.from_dynamo_item(item) for item in items] ) - return skills + return sorted(skills, key=lambda s: s.skill_id) except ClientError as e: logger.error(f"Error batch getting skills: {e}") diff --git a/backend/src/apis/shared/storage/dynamodb_storage.py b/backend/src/apis/shared/storage/dynamodb_storage.py index 3206c7227..077a8dffd 100644 --- a/backend/src/apis/shared/storage/dynamodb_storage.py +++ b/backend/src/apis/shared/storage/dynamodb_storage.py @@ -276,6 +276,54 @@ async def get_session_metadata( except ClientError as e: raise Exception(f"Failed to get session metadata: {e}") + async def get_session_cost_records( + self, + session_id: str + ) -> List[Dict[str, Any]]: + """ + Get ALL cost records for a session, chronologically, without a user + filter — the admin cost-anatomy view inspects any user's session. + Non-admin surfaces must keep using ``get_session_metadata`` (which + filters by userId). + + Schema: + GSI: SessionLookupIndex + GSI_PK: SESSION# + GSI_SK: begins_with C# (C# — chronological sort) + """ + from boto3.dynamodb.conditions import Key + + try: + items: List[Dict[str, Any]] = [] + last_evaluated_key = None + while True: + query_kwargs = { + "IndexName": "SessionLookupIndex", + "KeyConditionExpression": ( + Key("GSI_PK").eq(f"SESSION#{session_id}") + & Key("GSI_SK").begins_with("C#") + ), + "ScanIndexForward": True, + } + if last_evaluated_key: + query_kwargs["ExclusiveStartKey"] = last_evaluated_key + response = self.sessions_metadata_table.query(**query_kwargs) + items.extend(response.get("Items", [])) + last_evaluated_key = response.get("LastEvaluatedKey") + if not last_evaluated_key: + break + + results = [] + for item in items: + item_float = self._convert_decimal_to_float(item) + for key in ["PK", "SK", "GSI_PK", "GSI_SK", "GSI1PK", "GSI1SK", "ttl"]: + item_float.pop(key, None) + results.append(item_float) + return results + + except ClientError as e: + raise Exception(f"Failed to get session cost records: {e}") + async def get_user_cost_summary( self, user_id: str, diff --git a/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py b/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py new file mode 100644 index 000000000..4295cd3ac --- /dev/null +++ b/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py @@ -0,0 +1,217 @@ +""" +Bedrock prompt-cache resilience — three cachePoints per request. + +A message-level cache lookup can miss structurally: Anthropic's cache lookback +checks only ~20 content blocks behind the breakpoint, so a wide parallel tool +fan-out pushes the previous checkpoint out of range (prod session aecd387d: +cacheRead=0, cacheWrite=134k mid-turn). Dedicated cachePoints on toolConfig and +the system prompt keep the stable prefix readable from cache on those turns. + +Contract under test (see ModelConfig.to_bedrock_config comment): + 1. toolConfig.tools tail — via cache_tools="default" + 2. system tail — via SystemContentBlock list from AgentFactory + 3. last user message tail — via CacheConfig(strategy="auto") +Bedrock allows max 4 cachePoints per request; nothing else may add one, so the +formatted request must contain exactly 3. +""" + +import json +from unittest.mock import MagicMock, patch + +import pytest + +from agents.main_agent.core.model_config import ModelConfig, ModelProvider + +CLAUDE_MODEL_ID = "us.anthropic.claude-haiku-4-5-20251001-v1:0" + + +def _count_cache_points(node) -> int: + """Count every cachePoint block anywhere in a formatted request.""" + if isinstance(node, dict): + return sum(_count_cache_points(v) for v in node.values()) + ( + 1 if "cachePoint" in node else 0 + ) + if isinstance(node, list): + return sum(_count_cache_points(item) for item in node) + return 0 + + +# --------------------------------------------------------------------------- +# ModelConfig: cache_tools + support predicate +# --------------------------------------------------------------------------- +class TestCacheToolsConfig: + def test_cache_tools_set_for_claude_with_caching(self): + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True) + assert config.to_bedrock_config()["cache_tools"] == "default" + + def test_no_cache_tools_when_caching_disabled(self): + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=False) + assert "cache_tools" not in config.to_bedrock_config() + + def test_no_cache_tools_for_non_anthropic_bedrock_model(self): + """A model Strands' auto strategy would no-op on must not get explicit + cachePoints either — Bedrock would reject them with ValidationException.""" + config = ModelConfig(model_id="amazon.nova-pro-v1:0", caching_enabled=True) + assert "cache_tools" not in config.to_bedrock_config() + + def test_support_predicate_false_for_non_bedrock_provider(self): + config = ModelConfig( + model_id="gpt-4o", provider=ModelProvider.OPENAI, caching_enabled=True + ) + assert config.bedrock_cache_points_supported() is False + + +# --------------------------------------------------------------------------- +# AgentFactory: system prompt wrapped as SystemContentBlock list +# --------------------------------------------------------------------------- +class TestFactorySystemPromptCachePoint: + @patch("agents.main_agent.core.agent_factory.Agent") + @patch("agents.main_agent.core.agent_factory.CountTokensBedrockModel") + def test_system_prompt_gets_trailing_cache_point(self, _mock_model, mock_agent_cls): + from agents.main_agent.core.agent_factory import AgentFactory + + AgentFactory.create_agent( + model_config=ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True), + system_prompt="You are a helpful assistant.", + tools=[], + session_manager=MagicMock(), + ) + + assert mock_agent_cls.call_args.kwargs["system_prompt"] == [ + {"text": "You are a helpful assistant."}, + {"cachePoint": {"type": "default"}}, + ] + + @patch("agents.main_agent.core.agent_factory.Agent") + @patch("agents.main_agent.core.agent_factory.CountTokensBedrockModel") + def test_plain_string_when_caching_disabled(self, _mock_model, mock_agent_cls): + from agents.main_agent.core.agent_factory import AgentFactory + + AgentFactory.create_agent( + model_config=ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=False), + system_prompt="You are a helpful assistant.", + tools=[], + session_manager=MagicMock(), + ) + + assert ( + mock_agent_cls.call_args.kwargs["system_prompt"] + == "You are a helpful assistant." + ) + + @patch("agents.main_agent.core.agent_factory.Agent") + @patch("agents.main_agent.core.agent_factory.CountTokensBedrockModel") + def test_empty_prompt_never_wrapped(self, _mock_model, mock_agent_cls): + """Bedrock rejects a cachePoint with no preceding content.""" + from agents.main_agent.core.agent_factory import AgentFactory + + AgentFactory.create_agent( + model_config=ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True), + system_prompt="", + tools=[], + session_manager=MagicMock(), + ) + + assert mock_agent_cls.call_args.kwargs["system_prompt"] == "" + + +# --------------------------------------------------------------------------- +# End-to-end: the formatted ConverseStream request +# --------------------------------------------------------------------------- +class TestFormattedRequestCachePoints: + @pytest.fixture + def model(self, monkeypatch): + """Real Strands BedrockModel built from our production config path. + + boto3 client construction needs a region but no credentials, and + format_request never touches the network. + """ + monkeypatch.setenv("AWS_REGION", "us-west-2") + from agents.main_agent.core.bedrock_count_tokens import CountTokensBedrockModel + + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True) + return CountTokensBedrockModel(**config.to_bedrock_config()) + + @pytest.fixture + def request_parts(self, model): + """Format a fan-out-shaped conversation and return the request.""" + system_prompt_content = [ + {"text": "You are a helpful assistant."}, + {"cachePoint": {"type": "default"}}, + ] + tool_specs = [ + { + "name": "get_thread", + "description": "Fetch a thread", + "inputSchema": {"json": {"type": "object", "properties": {}}}, + } + ] + messages = [ + { + "role": "user", + # Stale message-level point from the previous turn — auto + # strategy must strip it (it manages message points itself). + "content": [{"text": "first turn"}, {"cachePoint": {"type": "default"}}], + }, + { + "role": "assistant", + "content": [ + {"toolUse": {"toolUseId": "t1", "name": "get_thread", "input": {}}} + ], + }, + { + "role": "user", + "content": [ + { + "toolResult": { + "toolUseId": "t1", + "content": [{"text": "thread body"}], + "status": "success", + } + } + ], + }, + ] + return model.format_request( + messages, tool_specs, system_prompt_content=system_prompt_content + ) + + def test_tool_config_tail_is_cache_point(self, request_parts): + assert request_parts["toolConfig"]["tools"][-1] == { + "cachePoint": {"type": "default"} + } + + def test_system_tail_is_cache_point(self, request_parts): + assert request_parts["system"][-1] == {"cachePoint": {"type": "default"}} + # Strands' auto strategy strips only message-level points — the + # system point must survive. + assert request_parts["system"][0] == {"text": "You are a helpful assistant."} + + def test_last_user_message_tail_is_cache_point(self, request_parts): + last_user = [m for m in request_parts["messages"] if m["role"] == "user"][-1] + assert last_user["content"][-1] == {"cachePoint": {"type": "default"}} + + def test_stale_message_cache_point_stripped(self, request_parts): + first_msg_blocks = request_parts["messages"][0]["content"] + assert all("cachePoint" not in block for block in first_msg_blocks) + + def test_exactly_three_cache_points_total(self, request_parts): + """Bedrock's hard limit is 4 cachePoints per request; we budget 3 + (tools, system, auto message point). If this fails at >3, something + new started adding cachePoints — rebalance the budget before shipping.""" + assert _count_cache_points(request_parts) == 3, json.dumps( + request_parts, default=str, indent=2 + ) + + def test_caching_disabled_yields_zero_cache_points(self, monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + from agents.main_agent.core.bedrock_count_tokens import CountTokensBedrockModel + + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=False) + model = CountTokensBedrockModel(**config.to_bedrock_config()) + request = model.format_request( + [{"role": "user", "content": [{"text": "hi"}]}], + None, + system_prompt_content=[{"text": "You are a helpful assistant."}], + ) + assert _count_cache_points(request) == 0 diff --git a/backend/tests/agents/main_agent/session/test_compaction_models.py b/backend/tests/agents/main_agent/session/test_compaction_models.py index 20f6ce634..898233625 100644 --- a/backend/tests/agents/main_agent/session/test_compaction_models.py +++ b/backend/tests/agents/main_agent/session/test_compaction_models.py @@ -49,6 +49,7 @@ def test_to_dict_has_camel_case_keys(self): "lastInputTokens", "updatedAt", "totalSummarizedTurns", + "truncationAnchor", } def test_to_dict_values_match(self): @@ -99,6 +100,17 @@ def test_from_dict_partial_data_fills_defaults(self): assert state.last_input_tokens == 0 assert state.updated_at is None + def test_from_dict_roundtrips_truncation_anchor(self): + state = CompactionState.from_dict({"checkpoint": 3, "truncationAnchor": 9}) + assert state.truncation_anchor == 9 + + def test_from_dict_legacy_record_defaults_anchor_to_checkpoint(self): + """Records written before the anchor existed must not re-truncate + retained history: the anchor defaults to the checkpoint, so nothing + the slice keeps is mutated on restore.""" + state = CompactionState.from_dict({"checkpoint": 7}) + assert state.truncation_anchor == 7 + # --------------------------------------------------------------------------- # CompactionState.from_dict with None or empty dict (Req 16.4) diff --git a/backend/tests/agents/main_agent/session/test_compaction_stability.py b/backend/tests/agents/main_agent/session/test_compaction_stability.py new file mode 100644 index 000000000..bc69080bd --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_compaction_stability.py @@ -0,0 +1,342 @@ +"""Byte-stability tests for restore-time compaction. + +Bedrock prompt caching requires an exact prefix match, so +``TurnBasedSessionManager.initialize()`` must derive the restored history as +a pure function of (stored messages, persisted compaction state). The old +design truncated tool contents behind a sliding protected-turns window, +which re-mutated the turn that just aged past the window on every restore — +breaking the cached prefix and forcing a full prefix re-write (~$2.5/MTok on +a 35k–150k prefix) nearly every turn (observed in prod session aecd387d: +-382/-1035/-1513 inter-turn prefix-token shrinkages with cacheRead=0 well +inside the cache TTL). + +These tests pin the redesign: truncation is driven only by the persisted +``truncation_anchor``, which moves at checkpoint advances (where the slice +already pays the one cache re-write) or opportunistically when the prompt +cache has already expired between turns. +""" + +import copy +import json +from datetime import datetime, timedelta, timezone +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session.compaction_models import CompactionConfig, CompactionState +from agents.main_agent.session.turn_based_session_manager import TurnBasedSessionManager + +from .conftest import ( + make_user_message, + make_assistant_message, + make_tool_use_message, + make_tool_result_message, +) + + +LONG_RESULT = "R" * 400 # well above the fixture's max_tool_content_length=50 + + +def make_tool_turn(i: int) -> list: + """One 4-message turn with a tool result long enough to be truncatable.""" + return [ + make_user_message(f"Question {i}"), + make_tool_use_message(f"t{i}", "search", {"q": f"query {i}"}), + make_tool_result_message(f"t{i}", LONG_RESULT), + make_assistant_message(f"Answer {i}"), + ] + + +def make_tool_conversation(turns: int) -> list: + messages = [] + for i in range(turns): + messages.extend(make_tool_turn(i)) + return messages + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +def _iso(dt: datetime) -> str: + return dt.isoformat() + + +def _dump(messages) -> str: + return json.dumps(messages, sort_keys=True) + + +def _fresh_state(**overrides) -> dict: + """Persisted state stamped 'just now' — the prompt cache is still warm.""" + return {"compaction": CompactionState(updated_at=_iso(_now()), **overrides).to_dict()} + + +def _stale_state(age_seconds: int = 600, **overrides) -> dict: + """Persisted state older than the cache TTL — the cache has expired.""" + return { + "compaction": CompactionState( + updated_at=_iso(_now() - timedelta(seconds=age_seconds)), **overrides + ).to_dict() + } + + +@pytest.fixture +def restore_session(make_session_manager, compaction_config): + """Simulate one full session restore against an in-memory state store. + + Each call builds a fresh manager (as a new container invocation would), + wires ``_load/_save_compaction_state`` to ``state_store`` so persistence + behaves like the sessions-metadata record, and runs the real + ``initialize()`` restore path (sanitize + compaction + repair). Returns + the derived ``agent.messages``. + """ + + def _restore(state_store: dict, stored_messages: list): + mgr = make_session_manager(compaction_config=compaction_config) + mgr._load_compaction_state = lambda: CompactionState.from_dict( + state_store.get("compaction") + ) + + def _save(state: CompactionState) -> None: + state.updated_at = _iso(_now()) + state_store["compaction"] = state.to_dict() + + mgr._save_compaction_state = _save + mgr._retrieve_session_summaries = lambda: [] + + session_agent = MagicMock() + session_agent.state = {} + session_agent.conversation_manager_state = {} + session_messages = [] + for msg in copy.deepcopy(stored_messages): + sm = MagicMock() + sm.to_message.return_value = msg + session_messages.append(sm) + + mgr.read_agent = MagicMock(return_value=session_agent) + mgr.list_messages = MagicMock(return_value=session_messages) + mgr._is_new_session = False + + agent = MagicMock() + agent.agent_id = "default" + agent.messages = [] + agent.conversation_manager.restore_from_session.return_value = [] + agent.conversation_manager.removed_message_count = 0 + + mgr.initialize(agent) + return agent.messages + + return _restore + + +@pytest.fixture +def make_wired_manager(make_session_manager, compaction_config): + """Manager wired to an in-memory state store, for update_after_turn tests.""" + + def _make(state_store: dict): + mgr = make_session_manager(compaction_config=compaction_config) + mgr._load_compaction_state = lambda: CompactionState.from_dict( + state_store.get("compaction") + ) + + def _save(state: CompactionState) -> None: + state.updated_at = _iso(_now()) + state_store["compaction"] = state.to_dict() + + mgr._save_compaction_state = _save + mgr._retrieve_session_summaries = lambda: [] + return mgr + + return _make + + +class TestByteStability: + """agent.messages must be byte-identical across consecutive restores + when no compaction-state change occurs.""" + + def test_consecutive_restores_identical_no_state(self, restore_session): + stored = make_tool_conversation(6) + store = {} + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + # Nothing may be truncated: no anchor has ever been set. + assert "[truncated" not in _dump(first) + + def test_consecutive_restores_identical_warm_cache(self, restore_session): + stored = make_tool_conversation(8) + store = _fresh_state() + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + assert "[truncated" not in _dump(first) + + def test_prefix_stable_as_turns_accumulate(self, restore_session): + """The regression the sliding window caused: appending a new turn must + not mutate any earlier message. Under the old design the turn that + aged past protected_turns was newly truncated here, breaking the + cached prefix on every turn.""" + stored = make_tool_conversation(6) + before = restore_session(_fresh_state(), stored) + + stored_next = stored + make_tool_turn(6) + # A real turn refreshes updated_at within the TTL. + after = restore_session(_fresh_state(), stored_next) + + assert _dump(after[: len(before)]) == _dump(before) + + def test_checkpointed_restore_is_stable(self, restore_session): + """With a persisted checkpoint + summary, consecutive restores still + derive the identical history (summary prepend is deterministic).""" + stored = make_tool_conversation(8) + checkpoint = 5 * 4 # start of turn 5 + store = _fresh_state( + checkpoint=checkpoint, + truncation_anchor=checkpoint, + summary="Earlier discussion about queries 0-4.", + ) + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + assert len(first) == len(stored) - checkpoint + assert "" in first[0]["content"][0]["text"] + assert "[truncated" not in _dump(first) + + def test_legacy_state_without_anchor_is_stable_and_untruncated(self, restore_session): + """Records written before truncationAnchor existed default the anchor + to the checkpoint: retained history is never truncated.""" + stored = make_tool_conversation(8) + legacy = { + "checkpoint": 20, + "summary": "Legacy summary", + "lastInputTokens": 120_000, + "updatedAt": _iso(_now()), + "totalSummarizedTurns": 5, + } + store = {"compaction": legacy} + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + assert "[truncated" not in _dump(first) + + +class TestAnchorTruncation: + def test_truncation_applies_only_below_anchor(self, restore_session): + stored = make_tool_conversation(6) + anchor = 3 * 4 # start of turn 3 + store = _fresh_state(truncation_anchor=anchor) + derived = restore_session(store, stored) + + assert "[truncated" in _dump(derived[:anchor]) + assert "[truncated" not in _dump(derived[anchor:]) + # And the derivation is stable. + assert _dump(restore_session(store, stored)) == _dump(derived) + + def test_anchor_beyond_history_is_clamped(self, restore_session): + stored = make_tool_conversation(2) + store = _fresh_state(truncation_anchor=999) + derived = restore_session(store, stored) + assert len(derived) == len(stored) + assert "[truncated" in _dump(derived) + + +class TestOpportunisticAnchorAdvance: + def test_expired_cache_advances_and_persists_anchor(self, restore_session): + """When the prompt cache has already expired (>TTL since the previous + turn) the anchor slides to the protected-turns boundary for free and + is persisted, so the very next restore derives identical history.""" + stored = make_tool_conversation(8) + store = _stale_state(age_seconds=600) + first = restore_session(store, stored) + + expected_anchor = (8 - 3) * 4 # cutoffs[-3] = start of turn 5 + assert store["compaction"]["truncationAnchor"] == expected_anchor + assert "[truncated" in _dump(first[:expected_anchor]) + assert "[truncated" not in _dump(first[expected_anchor:]) + + # The save stamped updated_at=now, so the follow-up restore is inside + # the TTL: no further movement, byte-identical output. + second = restore_session(store, stored) + assert store["compaction"]["truncationAnchor"] == expected_anchor + assert _dump(first) == _dump(second) + + def test_warm_cache_never_advances_anchor(self, restore_session): + stored = make_tool_conversation(8) + store = _fresh_state() + restore_session(store, stored) + assert store["compaction"]["truncationAnchor"] == 0 + + def test_no_advance_when_too_few_turns(self, restore_session): + stored = make_tool_conversation(3) + store = _stale_state(age_seconds=600) + derived = restore_session(store, stored) + assert store["compaction"]["truncationAnchor"] == 0 + assert "[truncated" not in _dump(derived) + + def test_missing_updated_at_treated_as_warm(self, restore_session): + stored = make_tool_conversation(8) + store = {"compaction": CompactionState().to_dict()} # updated_at=None + restore_session(store, stored) + assert store["compaction"]["truncationAnchor"] == 0 + + +class TestCacheWindowExpired: + def test_none_and_garbage_are_not_expired(self): + assert TurnBasedSessionManager._cache_window_expired(None, 300) is False + assert TurnBasedSessionManager._cache_window_expired("not-a-timestamp", 300) is False + + def test_z_suffix_and_naive_timestamps_parse(self): + old = (_now() - timedelta(seconds=900)).strftime("%Y-%m-%dT%H:%M:%S") + assert TurnBasedSessionManager._cache_window_expired(old + "Z", 300) is True + assert TurnBasedSessionManager._cache_window_expired(old, 300) is True # naive → UTC + + def test_recent_timestamp_not_expired(self): + assert TurnBasedSessionManager._cache_window_expired(_iso(_now()), 300) is False + + +class TestCheckpointAdvanceMovesAnchor: + @pytest.mark.asyncio + async def test_update_after_turn_sets_anchor_with_checkpoint( + self, make_wired_manager, restore_session + ): + stored = make_tool_conversation(8) + store = _fresh_state() + mgr = make_wired_manager(store) + + result = await mgr.update_after_turn(input_tokens=5000, current_messages=stored) + + expected_checkpoint = (8 - 3) * 4 + assert result is not None + assert result.new_checkpoint == expected_checkpoint + assert mgr.compaction_state.checkpoint == expected_checkpoint + assert mgr.compaction_state.truncation_anchor == expected_checkpoint + assert store["compaction"]["truncationAnchor"] == expected_checkpoint + + # Post-advance restores are byte-stable and truncate nothing the + # slice retained (anchor == checkpoint). + first = restore_session(store, stored) + second = restore_session(store, stored) + assert _dump(first) == _dump(second) + assert "[truncated" not in _dump(first) + + @pytest.mark.asyncio + async def test_anchor_never_regresses_below_prior_anchor(self, make_wired_manager): + stored = make_tool_conversation(8) + anchor = (8 - 3) * 4 + 4 # ahead of where the checkpoint will land + store = _fresh_state(truncation_anchor=anchor) + mgr = make_wired_manager(store) + + result = await mgr.update_after_turn(input_tokens=5000, current_messages=stored) + assert result is not None + assert mgr.compaction_state.truncation_anchor == anchor + + @pytest.mark.asyncio + async def test_below_threshold_saves_state_without_movement(self, make_wired_manager): + stored = make_tool_conversation(8) + store = _fresh_state(truncation_anchor=8, checkpoint=4) + mgr = make_wired_manager(store) + + result = await mgr.update_after_turn(input_tokens=10, current_messages=stored) + assert result is None + assert store["compaction"]["checkpoint"] == 4 + assert store["compaction"]["truncationAnchor"] == 8 diff --git a/backend/tests/agents/main_agent/session/test_prefix_fingerprint_hook.py b/backend/tests/agents/main_agent/session/test_prefix_fingerprint_hook.py new file mode 100644 index 000000000..f4e1471c3 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_prefix_fingerprint_hook.py @@ -0,0 +1,126 @@ +"""PrefixFingerprintHook — per-model-call prompt-cache prefix hashes.""" + +from types import SimpleNamespace + +from agents.main_agent.session.hooks.prefix_fingerprint import ( + PrefixFingerprintHook, + get_prefix_fingerprint, + reset_prefix_fingerprints, +) + + +def _agent(tool_specs=None, system_prompt="You are helpful.", messages=None): + return SimpleNamespace( + tool_registry=SimpleNamespace( + get_all_tool_specs=lambda: tool_specs if tool_specs is not None else [] + ), + system_prompt=system_prompt, + messages=messages if messages is not None else [], + ) + + +def _fire(agent): + PrefixFingerprintHook()._on_before_model_call(SimpleNamespace(agent=agent)) + + +def _msg(role, text): + return {"role": role, "content": [{"text": text}]} + + +class TestFingerprintCapture: + def test_appends_one_entry_per_model_call(self): + agent = _agent(messages=[_msg("user", "hi")]) + reset_prefix_fingerprints(agent) + _fire(agent) + agent.messages.append(_msg("assistant", "hello")) + agent.messages.append(_msg("user", "more")) + _fire(agent) + + first = get_prefix_fingerprint(agent, 0) + second = get_prefix_fingerprint(agent, 1) + assert first is not None and second is not None + assert first["messageCount"] == 1 + assert second["messageCount"] == 3 + # Latest-entry fallback used by single-call persistence paths. + assert get_prefix_fingerprint(agent, None) == second + + def test_fingerprint_shape(self): + agent = _agent(tool_specs=[{"name": "t1"}], messages=[_msg("user", "hi")]) + reset_prefix_fingerprints(agent) + _fire(agent) + fp = get_prefix_fingerprint(agent, 0) + assert set(fp) == { + "toolConfigHash", + "systemPromptHash", + "historyHash", + "messageCount", + } + assert all(len(fp[k]) == 16 for k in ("toolConfigHash", "systemPromptHash", "historyHash")) + + def test_history_hash_excludes_newest_message(self): + # Two calls whose histories share the same prefix but different + # newest message must produce the SAME historyHash: for a cache hit + # the prior prefix is what must match. + shared_prefix = [_msg("user", "hi"), _msg("assistant", "hello")] + a = _agent(messages=shared_prefix + [_msg("user", "question A")]) + b = _agent(messages=shared_prefix + [_msg("user", "question B")]) + reset_prefix_fingerprints(a) + reset_prefix_fingerprints(b) + _fire(a) + _fire(b) + assert ( + get_prefix_fingerprint(a, 0)["historyHash"] + == get_prefix_fingerprint(b, 0)["historyHash"] + ) + + def test_tool_order_flip_changes_tool_config_hash(self): + specs = [{"name": "alpha"}, {"name": "beta"}] + a = _agent(tool_specs=list(specs)) + b = _agent(tool_specs=list(reversed(specs))) + reset_prefix_fingerprints(a) + reset_prefix_fingerprints(b) + _fire(a) + _fire(b) + assert ( + get_prefix_fingerprint(a, 0)["toolConfigHash"] + != get_prefix_fingerprint(b, 0)["toolConfigHash"] + ) + + def test_structured_system_prompt_supported(self): + # AgentSkills' block-level injection can turn system_prompt into a + # list of SystemContentBlock dicts. + agent = _agent(system_prompt=[{"text": "base"}, {"text": ""}]) + reset_prefix_fingerprints(agent) + _fire(agent) + assert get_prefix_fingerprint(agent, 0)["systemPromptHash"] + + +class TestLifecycleAndSafety: + def test_reset_clears_previous_turn(self): + agent = _agent() + reset_prefix_fingerprints(agent) + _fire(agent) + reset_prefix_fingerprints(agent) + assert get_prefix_fingerprint(agent, 0) is None + assert get_prefix_fingerprint(agent, None) is None + + def test_out_of_range_index_returns_none(self): + agent = _agent() + reset_prefix_fingerprints(agent) + _fire(agent) + assert get_prefix_fingerprint(agent, 5) is None + + def test_never_raises_on_broken_agent(self): + class _Broken: + @property + def tool_registry(self): + raise RuntimeError("boom") + + broken = _Broken() + _fire(broken) # must swallow + assert get_prefix_fingerprint(broken, None) is None + + def test_works_without_prior_reset(self): + agent = _agent() + _fire(agent) + assert get_prefix_fingerprint(agent, 0) is not None diff --git a/backend/tests/agents/main_agent/session/test_turn_based_session_manager.py b/backend/tests/agents/main_agent/session/test_turn_based_session_manager.py index efd908e57..77e111550 100644 --- a/backend/tests/agents/main_agent/session/test_turn_based_session_manager.py +++ b/backend/tests/agents/main_agent/session/test_turn_based_session_manager.py @@ -130,33 +130,6 @@ def test_only_assistant_messages(self, make_session_manager): assert mgr._find_valid_cutoff_indices(messages) == [] -class TestFindProtectedIndices: - - def test_protect_last_2_turns(self, make_session_manager): - mgr = make_session_manager() - messages = make_conversation(4) # indices 0-7, turns at 0,2,4,6 - protected = mgr._find_protected_indices(messages, 2) - # Last 2 turn starts are at index 4 and 6, so protect 4..7 - assert protected == set(range(4, 8)) - - def test_zero_protected_turns(self, make_session_manager): - mgr = make_session_manager() - messages = make_conversation(3) - assert mgr._find_protected_indices(messages, 0) == set() - - def test_more_protected_than_available(self, make_session_manager): - mgr = make_session_manager() - messages = make_conversation(2) # 4 messages, 2 turns - protected = mgr._find_protected_indices(messages, 10) - # All messages protected since we only have 2 turns - assert protected == set(range(0, 4)) - - def test_no_valid_cutoffs(self, make_session_manager): - mgr = make_session_manager() - messages = [make_assistant_message("a1")] - assert mgr._find_protected_indices(messages, 2) == set() - - # =========================================================================== # Task 3 — Tool content truncation (Stage 1) # =========================================================================== @@ -589,19 +562,21 @@ def test_existing_agent_no_compaction_loads_all_messages(self, make_session_mana assert mgr.message_count == 6 def test_existing_agent_compaction_no_checkpoint(self, make_session_manager, compaction_config): - """Compaction enabled with checkpoint=0 should load all messages with truncation.""" + """Compaction enabled with checkpoint=0 loads all messages untouched: + with no truncation anchor set, even a long tool result is preserved + byte-for-byte (truncation is anchor-driven, never per-restore).""" mgr = make_session_manager(compaction_config=compaction_config) mgr._load_compaction_state = MagicMock(return_value=CompactionState()) - # A valid Converse history: the truncatable toolResult is preceded by - # its matching toolUse turn, so the restore-time pairing repair no-ops - # and this test exercises compaction/truncation in isolation. + # A valid Converse history: the toolResult is preceded by its matching + # toolUse turn, so the restore-time pairing repair no-ops and this + # test exercises compaction in isolation. messages = [ make_user_message("q1"), make_assistant_message("a1"), make_user_message("q2"), make_tool_use_message("t1", "search", {"q": "x"}), - make_tool_result_message("t1", "x" * 200), # will be truncated + make_tool_result_message("t1", "x" * 200), ] session_agent = self._make_mock_session_agent() mgr.read_agent = MagicMock(return_value=session_agent) @@ -611,8 +586,9 @@ def test_existing_agent_compaction_no_checkpoint(self, make_session_manager, com agent = self._make_mock_agent() mgr.initialize(agent) - # All 5 messages kept (checkpoint=0), but truncation applied + # All 5 messages kept (checkpoint=0) with no truncation (anchor=0) assert len(agent.messages) == 5 + assert agent.messages[4]["content"][0]["toolResult"]["content"][0]["text"] == "x" * 200 # Valid cutoffs cached for user text messages (indices 0, 2); the # toolResult user turn at index 4 is not a valid cutoff. assert mgr._valid_cutoff_indices == [0, 2] @@ -641,10 +617,12 @@ def test_existing_agent_compaction_with_checkpoint_slices_messages(self, make_se assert "" in first_text def test_existing_agent_compaction_checkpoint_plus_truncation(self, make_session_manager, compaction_config): - """Both checkpoint slicing and truncation should apply.""" + """Checkpoint slicing plus anchor-driven truncation: messages between + the checkpoint and the persisted truncation anchor are truncated; + messages at or after the anchor are preserved byte-for-byte.""" mgr = make_session_manager(compaction_config=compaction_config) mgr._load_compaction_state = MagicMock( - return_value=CompactionState(checkpoint=2) + return_value=CompactionState(checkpoint=2, truncation_anchor=5) ) # Valid Converse history: the truncatable toolResult follows its @@ -655,7 +633,11 @@ def test_existing_agent_compaction_checkpoint_plus_truncation(self, make_session make_assistant_message("old2"), make_user_message("new1"), make_tool_use_message("t1", "search", {"q": "r"}), - make_tool_result_message("t1", "r" * 200), # truncatable + make_tool_result_message("t1", "r" * 200), # below anchor: truncated + make_assistant_message("answer1"), + make_user_message("new2"), + make_tool_use_message("t2", "search", {"q": "s"}), + make_tool_result_message("t2", "s" * 200), # at/after anchor: preserved ] session_agent = self._make_mock_session_agent() mgr.read_agent = MagicMock(return_value=session_agent) @@ -665,8 +647,13 @@ def test_existing_agent_compaction_checkpoint_plus_truncation(self, make_session agent = self._make_mock_agent() mgr.initialize(agent) - # Sliced from index 2: [new1, toolUse, toolResult] = 3 messages remain - assert len(agent.messages) == 3 + # Sliced from index 2: 7 messages remain + assert len(agent.messages) == 7 + # Absolute index 4 (below anchor 5) truncated to max_tool_content_length=50 + truncated_text = agent.messages[2]["content"][0]["toolResult"]["content"][0]["text"] + assert "[truncated" in truncated_text + # Absolute index 8 (after anchor) untouched + assert agent.messages[6]["content"][0]["toolResult"]["content"][0]["text"] == "s" * 200 def test_duplicate_agent_id_raises(self, make_session_manager): """Second initialize with same agent_id should raise SessionException.""" diff --git a/backend/tests/agents/main_agent/skills/test_strands_mapping.py b/backend/tests/agents/main_agent/skills/test_strands_mapping.py index 1e2b427d9..85b7ff978 100644 --- a/backend/tests/agents/main_agent/skills/test_strands_mapping.py +++ b/backend/tests/agents/main_agent/skills/test_strands_mapping.py @@ -130,6 +130,28 @@ def test_returns_plugin_and_tool(self, monkeypatch): assert isinstance(plugin, AgentSkills) assert read_tool.tool_name == "read_skill_file" + def test_prompt_order_stable_across_fetch_order(self, monkeypatch): + # The plugin renders in the order it receives the + # skills; a between-turn order flip changes the system prompt and + # invalidates the Bedrock prompt cache. Regardless of the order the + # fetch returns records, the runtime must produce one canonical order. + recs = [ + _record(skill_id="zeta_skill"), + _record(skill_id="alpha_skill"), + _record(skill_id="midway_skill"), + ] + + def names_for(fetch_order): + monkeypatch.setattr( + sm, "fetch_active_skill_records", lambda ids: list(fetch_order) + ) + plugin, _ = sm.build_skills_runtime(["any"]) + return [s.name for s in plugin.get_available_skills()] + + forward = names_for(recs) + reversed_order = names_for(list(reversed(recs))) + assert forward == reversed_order == ["alpha-skill", "midway-skill", "zeta-skill"] + class TestReadSkillFile: def _tool(self, monkeypatch, records, store): diff --git a/backend/tests/agents/main_agent/test_prompt_cache_determinism.py b/backend/tests/agents/main_agent/test_prompt_cache_determinism.py new file mode 100644 index 000000000..b4f09e326 --- /dev/null +++ b/backend/tests/agents/main_agent/test_prompt_cache_determinism.py @@ -0,0 +1,173 @@ +"""CI determinism guard for the Bedrock prompt-cache prefix. + +Bedrock prompt caching is exact-prefix-match: if the system prompt or +toolConfig differs between two turns of the same session, the cache is +re-written at a premium. A prod audit (session aecd387d, $1.60) traced 75% +of spend to exactly that — skill records reaching the ```` +block in nondeterministic order, and RBAC grant unions materialized from +sets (hash-randomized iteration order). + +This guard builds the chat-agent surface twice from *shuffled* skill and +tool record orders and asserts the production prefix fingerprints +(``PrefixFingerprintHook`` — the same hashes persisted on every cost row) +are identical. It complements the targeted sorting fixes in +``apis.shared.skills.repository``, ``apis.shared.rbac.service``, and +``agents.main_agent.skills.strands_mapping``: any future assembly stage +that reintroduces order instability fails here. +""" + +import random +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest +from strands import Agent, tool +from strands.hooks import BeforeInvocationEvent + +from agents.main_agent.session.hooks.prefix_fingerprint import ( + PrefixFingerprintHook, + get_prefix_fingerprint, + reset_prefix_fingerprints, +) +from agents.main_agent.skills import strands_mapping as sm +from apis.shared.rbac.service import AppRoleService + + +# --------------------------------------------------------------------------- +# Fixture data: skill records + local tools +# --------------------------------------------------------------------------- + +def _skill_record(skill_id: str) -> SimpleNamespace: + return SimpleNamespace( + skill_id=skill_id, + display_name=skill_id.replace("_", " ").title(), + description=f"Description for {skill_id}.", + instructions=f"# {skill_id}\nInstructions body.", + allowed_tools=[], + skill_metadata={}, + resources=[], + status="active", + ) + + +SKILL_RECORDS = [ + _skill_record("zeta_skill"), + _skill_record("alpha_skill"), + _skill_record("midway_skill"), + _skill_record("beta_skill"), +] + + +@tool +def alpha_tool(query: str) -> str: + """Alpha tool.""" + return query + + +@tool +def beta_tool(query: str) -> str: + """Beta tool.""" + return query + + +@tool +def gamma_tool(query: str) -> str: + """Gamma tool.""" + return query + + +TOOLS_BY_ID = {"alpha_tool": alpha_tool, "beta_tool": beta_tool, "gamma_tool": gamma_tool} + + +def _role(role_id: str, tools: list, skills: list, priority: int = 10) -> SimpleNamespace: + return SimpleNamespace( + role_id=role_id, + priority=priority, + effective_permissions=SimpleNamespace( + tools=tools, models=[], skills=skills, quota_tier=None + ), + ) + + +# --------------------------------------------------------------------------- +# Agent assembly mirroring the production pipeline stages under test: +# RBAC merge (grant union → enabled ids) → skills runtime (records → +# AgentSkills plugin) → Agent construction → plugin system-prompt injection +# → PrefixFingerprintHook (the persisted hashes). +# --------------------------------------------------------------------------- + +async def _build_and_fingerprint(monkeypatch, rng: random.Random) -> dict: + # RBAC seam: roles and their granted lists in shuffled order — the union + # used to be materialized from a set, so order leaked hash randomization. + role_a_tools = ["alpha_tool", "beta_tool"] + role_b_tools = ["gamma_tool", "beta_tool"] + role_skills = [r.skill_id for r in SKILL_RECORDS] + rng.shuffle(role_a_tools) + rng.shuffle(role_b_tools) + rng.shuffle(role_skills) + roles = [ + _role("role-a", role_a_tools, role_skills[:2]), + _role("role-b", role_b_tools, role_skills[2:]), + ] + rng.shuffle(roles) + merged = AppRoleService._merge_permissions(None, user_id="user-1", roles=roles) + + # Skills seam: fetch returns records in shuffled (nondeterministic) order. + shuffled_records = list(SKILL_RECORDS) + rng.shuffle(shuffled_records) + monkeypatch.setattr(sm, "fetch_active_skill_records", lambda ids: shuffled_records) + plugin, read_tool = sm.build_skills_runtime(merged.skills) + + tools = [TOOLS_BY_ID[tool_id] for tool_id in merged.tools if tool_id in TOOLS_BY_ID] + tools.append(read_tool) + + model = MagicMock() + model.config = {} + agent = Agent( + model=model, + system_prompt="You are the campus assistant.", + tools=tools, + plugins=[plugin], + ) + + # Drive the plugin's system-prompt injection the way a real invocation + # does, then capture the same fingerprints production persists per call. + await plugin._on_before_invocation(BeforeInvocationEvent(agent=agent)) + reset_prefix_fingerprints(agent) + PrefixFingerprintHook()._on_before_model_call(SimpleNamespace(agent=agent)) + + fingerprint = get_prefix_fingerprint(agent, 0) + assert fingerprint is not None, "fingerprint hook failed to capture" + return fingerprint + + +@pytest.mark.asyncio +async def test_agent_prefix_identical_across_shuffled_record_orders(monkeypatch): + fp_a = await _build_and_fingerprint(monkeypatch, random.Random(1234)) + fp_b = await _build_and_fingerprint(monkeypatch, random.Random(987654)) + + assert fp_a["systemPromptHash"] == fp_b["systemPromptHash"], ( + "System prompt differs between two builds from shuffled skill/role " + "record orders — a nondeterministic assembly stage is back and will " + "invalidate the Bedrock prompt cache every turn." + ) + assert fp_a["toolConfigHash"] == fp_b["toolConfigHash"], ( + "toolConfig differs between two builds from shuffled tool-grant " + "orders — a nondeterministic assembly stage is back and will " + "invalidate the Bedrock prompt cache every turn." + ) + + +@pytest.mark.asyncio +async def test_skills_xml_actually_present_in_fingerprinted_prompt(monkeypatch): + # Guard the guard: if AgentSkills ever stops injecting before the hook + # captures, the system-prompt assertion above would trivially pass. + monkeypatch.setattr(sm, "fetch_active_skill_records", lambda ids: list(SKILL_RECORDS)) + plugin, read_tool = sm.build_skills_runtime([r.skill_id for r in SKILL_RECORDS]) + model = MagicMock() + model.config = {} + agent = Agent(model=model, system_prompt="Base.", tools=[read_tool], plugins=[plugin]) + await plugin._on_before_invocation(BeforeInvocationEvent(agent=agent)) + prompt_text = str(agent.system_prompt) + assert "available_skills" in prompt_text + assert "alpha-skill" in prompt_text diff --git a/backend/tests/shared/test_metadata_cache_derivation.py b/backend/tests/shared/test_metadata_cache_derivation.py new file mode 100644 index 000000000..7f8da4f63 --- /dev/null +++ b/backend/tests/shared/test_metadata_cache_derivation.py @@ -0,0 +1,245 @@ +"""Write-time cache derivation in apis.shared.sessions.metadata: +``_derive_cache_observability`` (previous-row classification) and the +cache-efficiency counters ``_bump_session_aggregates`` adds to the session row. +""" + +from datetime import datetime, timedelta, timezone +from decimal import Decimal + +import pytest + +from apis.shared.sessions import metadata as md +from apis.shared.sessions.models import ( + MessageMetadata, + ModelInfo, + PricingSnapshot, + TokenUsage, +) + + +NOW = datetime(2026, 7, 19, 12, 0, 0, tzinfo=timezone.utc) + + +def _metadata(input_t=100, output_t=50, cache_read=0, cache_write=0, with_pricing=True): + pricing = None + if with_pricing: + pricing = PricingSnapshot( + input_price_per_mtok=3.0, + output_price_per_mtok=15.0, + cache_write_price_per_mtok=3.75, + cache_read_price_per_mtok=0.30, + snapshot_at=NOW.isoformat(), + ) + return MessageMetadata( + token_usage=TokenUsage( + input_tokens=input_t, + output_tokens=output_t, + total_tokens=input_t + output_t, + cache_read_input_tokens=cache_read, + cache_write_input_tokens=cache_write, + ), + model_info=ModelInfo( + model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0", + model_name="Claude Sonnet 4.5", + pricing_snapshot=pricing, + ), + ) + + +class _FakeTable: + """Fake DynamoDB table: query() serves the previous-cost-row lookup.""" + + def __init__(self, prev_items=None): + self._prev_items = prev_items or [] + self.query_kwargs = None + self.update_kwargs = None + + def query(self, **kwargs): + self.query_kwargs = kwargs + return {"Items": self._prev_items} + + def update_item(self, **kwargs): + self.update_kwargs = kwargs + return {} + + +def _prev_row(seconds_ago, cache_read=0, cache_write=0): + ts = (NOW - timedelta(seconds=seconds_ago)).isoformat() + return { + "timestamp": ts, + "tokenUsage": { + "cacheReadInputTokens": Decimal(cache_read), + "cacheWriteInputTokens": Decimal(cache_write), + }, + } + + +class TestDeriveCacheObservability: + def _derive(self, table, meta): + return md._derive_cache_observability( + session_id="sess-1", + table=table, + timestamp=NOW.isoformat(), + message_metadata=meta, + ) + + def test_no_token_usage_returns_empty(self): + meta = MessageMetadata(token_usage=None) + assert self._derive(_FakeTable(), meta) == {} + + def test_first_write_without_previous_row(self): + result = self._derive(_FakeTable(), _metadata(cache_write=5000)) + assert result["cacheStatus"] == "first_write" + assert result["wastedUsd"] == 0.0 + assert "cacheGapSeconds" not in result + + def test_hit_with_previous_row(self): + table = _FakeTable([_prev_row(seconds_ago=30, cache_write=5000)]) + result = self._derive(table, _metadata(cache_read=5000, cache_write=100)) + assert result["cacheStatus"] == "hit" + assert result["cacheGapSeconds"] == 30 + assert result["wastedUsd"] == 0.0 + + def test_avoidable_miss_within_ttl_prices_waste(self): + # Previous call cached 4000 + 1000 tokens; this call re-writes 6000 + # with zero read, 60s later → avoidable. Waste = min(6000, 5000) + # tokens at the (3.75 - 0.30)/Mtok premium. + table = _FakeTable([_prev_row(seconds_ago=60, cache_read=4000, cache_write=1000)]) + result = self._derive(table, _metadata(cache_write=6000)) + assert result["cacheStatus"] == "miss_avoidable" + assert result["cacheGapSeconds"] == 60 + assert result["wastedUsd"] == round((5000 / 1_000_000) * 3.45, 6) + + def test_ttl_expired_miss_beyond_gap(self): + table = _FakeTable([_prev_row(seconds_ago=1200, cache_write=5000)]) + result = self._derive(table, _metadata(cache_write=6000)) + assert result["cacheStatus"] == "miss_ttl_expired" + assert result["wastedUsd"] == 0.0 + + def test_uncached_when_no_cache_activity(self): + table = _FakeTable([_prev_row(seconds_ago=30)]) + result = self._derive(table, _metadata()) + assert result["cacheStatus"] == "uncached" + + def test_first_write_after_below_threshold_uncached_call(self): + # Session whose prior calls were all below the minimum cacheable + # prefix: the previous row shows zero cache activity, so this call's + # write (crossing the threshold) is the expected first population — + # not miss_avoidable, and never priced as waste. + table = _FakeTable([_prev_row(seconds_ago=30)]) + result = self._derive(table, _metadata(cache_write=4122)) + assert result["cacheStatus"] == "first_write" + assert result["wastedUsd"] == 0.0 + assert result["cacheGapSeconds"] == 30 + + def test_query_targets_previous_row_descending_limit_1(self): + table = _FakeTable() + self._derive(table, _metadata(cache_write=100)) + assert table.query_kwargs["IndexName"] == "SessionLookupIndex" + assert table.query_kwargs["ScanIndexForward"] is False + assert table.query_kwargs["Limit"] == 1 + + def test_failure_returns_empty_never_raises(self): + class _Boom: + def query(self, **kwargs): + raise RuntimeError("dynamo down") + + assert self._derive(_Boom(), _metadata(cache_write=100)) == {} + + +class TestBumpSessionAggregatesCacheCounters: + @pytest.fixture + def session_lookup(self, monkeypatch): + async def _fake_get_session(session_id, user_id, table): + return {"SK": f"S#{session_id}"} + + monkeypatch.setattr(md, "_get_session_by_gsi", _fake_get_session) + + async def _bump(self, table, meta, observability): + await md._bump_session_aggregates( + session_id="sess-1", + user_id="user-1", + message_metadata=meta, + table=table, + cache_observability=observability, + ) + + @pytest.mark.asyncio + async def test_adds_cache_counters_next_to_total_cost(self, session_lookup): + table = _FakeTable() + meta = _metadata(cache_read=4000, cache_write=1000) + await self._bump( + table, meta, {"cacheStatus": "miss_avoidable", "wastedUsd": 0.01725} + ) + + expr = table.update_kwargs["UpdateExpression"] + values = table.update_kwargs["ExpressionAttributeValues"] + for attr in ( + "totalCost :c", + "totalCacheReadTokens :cacheRead", + "totalCacheWriteTokens :cacheWrite", + "avoidableMissCount :avoidableMiss", + "wastedUsd :wasted", + ): + assert attr in expr + assert values[":cacheRead"] == 4000 + assert values[":cacheWrite"] == 1000 + assert values[":avoidableMiss"] == 1 + assert values[":wasted"] == Decimal("0.01725") + + @pytest.mark.asyncio + async def test_zero_deltas_when_not_avoidable(self, session_lookup): + table = _FakeTable() + await self._bump(table, _metadata(cache_read=4000), {"cacheStatus": "hit", "wastedUsd": 0.0}) + values = table.update_kwargs["ExpressionAttributeValues"] + assert values[":avoidableMiss"] == 0 + assert values[":wasted"] == Decimal("0") + + @pytest.mark.asyncio + async def test_missing_observability_defaults_to_zeroes(self, session_lookup): + table = _FakeTable() + await self._bump(table, _metadata(), None) + values = table.update_kwargs["ExpressionAttributeValues"] + assert values[":cacheRead"] == 0 + assert values[":cacheWrite"] == 0 + assert values[":avoidableMiss"] == 0 + + +class TestKillSwitch: + """PROMPT_CACHE_OBSERVABILITY_ENABLED=false disables derivation and EMF.""" + + def test_flag_default_and_parsing(self, monkeypatch): + from apis.shared.observability import ( + PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, + prompt_cache_observability_enabled, + ) + + monkeypatch.delenv(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, raising=False) + assert prompt_cache_observability_enabled() is True + # Empty string (workflow env vars can materialize as "") stays enabled + monkeypatch.setenv(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, "") + assert prompt_cache_observability_enabled() is True + monkeypatch.setenv(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, "FALSE") + assert prompt_cache_observability_enabled() is False + monkeypatch.setenv(PROMPT_CACHE_OBSERVABILITY_ENABLED_ENV, "true") + assert prompt_cache_observability_enabled() is True + + def test_derivation_disabled_returns_empty(self, monkeypatch): + monkeypatch.setenv("PROMPT_CACHE_OBSERVABILITY_ENABLED", "false") + table = _FakeTable([_prev_row(seconds_ago=30, cache_write=5000)]) + result = md._derive_cache_observability( + session_id="sess-1", + table=table, + timestamp=NOW.isoformat(), + message_metadata=_metadata(cache_read=5000, cache_write=100), + ) + assert result == {} + + def test_emf_disabled_emits_nothing(self, monkeypatch, capsys): + monkeypatch.setenv("PROMPT_CACHE_OBSERVABILITY_ENABLED", "false") + md._emit_cache_metrics( + session_id="sess-1", + message_metadata=_metadata(cache_read=1000, cache_write=200), + cache_observability={"cacheStatus": "miss_avoidable", "wastedUsd": 0.01}, + ) + assert capsys.readouterr().out == "" diff --git a/backend/tests/shared/test_prompt_cache_observability.py b/backend/tests/shared/test_prompt_cache_observability.py new file mode 100644 index 000000000..43523276e --- /dev/null +++ b/backend/tests/shared/test_prompt_cache_observability.py @@ -0,0 +1,235 @@ +"""Prompt-cache observability primitives: classification, waste pricing, +prefix fingerprints, and EMF record shape.""" + +import io +import json +import logging + +from apis.shared.observability import ( + CACHE_TTL_SECONDS, + CacheStatus, + classify_cache_status, + compute_wasted_usd, + emit_prompt_cache_metrics, + fingerprint_canonical_json, + fingerprint_text, +) + + +PRICING = { + "inputPricePerMtok": 3.0, + "outputPricePerMtok": 15.0, + "cacheWritePricePerMtok": 3.75, + "cacheReadPricePerMtok": 0.30, +} + + +class TestClassifyCacheStatus: + def test_first_write(self): + assert ( + classify_cache_status(0, 5000, previous_call_exists=False, gap_seconds=None) + is CacheStatus.FIRST_WRITE + ) + + def test_hit_when_any_cache_read(self): + assert ( + classify_cache_status(4000, 200, previous_call_exists=True, gap_seconds=10) + is CacheStatus.HIT + ) + + def test_hit_wins_even_without_previous_row(self): + # A read implies the cache was warm regardless of what rows we kept. + assert ( + classify_cache_status(4000, 0, previous_call_exists=False, gap_seconds=None) + is CacheStatus.HIT + ) + + def test_miss_avoidable_within_ttl(self): + assert ( + classify_cache_status(0, 5000, previous_call_exists=True, gap_seconds=45) + is CacheStatus.MISS_AVOIDABLE + ) + + def test_miss_ttl_expired_beyond_ttl(self): + assert ( + classify_cache_status( + 0, 5000, previous_call_exists=True, gap_seconds=CACHE_TTL_SECONDS + 1 + ) + is CacheStatus.MISS_TTL_EXPIRED + ) + + def test_boundary_gap_exactly_ttl_is_avoidable(self): + assert ( + classify_cache_status( + 0, 5000, previous_call_exists=True, gap_seconds=CACHE_TTL_SECONDS + ) + is CacheStatus.MISS_AVOIDABLE + ) + + def test_unknown_gap_is_conservatively_expired(self): + assert ( + classify_cache_status(0, 5000, previous_call_exists=True, gap_seconds=None) + is CacheStatus.MISS_TTL_EXPIRED + ) + + def test_uncached_when_no_cache_activity(self): + assert ( + classify_cache_status(0, 0, previous_call_exists=True, gap_seconds=10) + is CacheStatus.UNCACHED + ) + + def test_below_threshold_then_crossing_is_first_write(self): + # Prior calls were below the minimum cacheable prefix (uncached), so + # no cache entry existed — the first call that crosses the threshold + # is the expected initial population, not an avoidable miss. + assert ( + classify_cache_status( + 0, + 4122, + previous_call_exists=True, + gap_seconds=45, + previous_cached_prefix_tokens=0, + ) + is CacheStatus.FIRST_WRITE + ) + + def test_unknown_previous_prefix_stays_avoidable(self): + # None means we couldn't see the previous call's cache split — keep + # the pre-existing (avoidable) classification rather than masking. + assert ( + classify_cache_status( + 0, + 5000, + previous_call_exists=True, + gap_seconds=45, + previous_cached_prefix_tokens=None, + ) + is CacheStatus.MISS_AVOIDABLE + ) + + +class TestComputeWastedUsd: + def test_avoidable_miss_priced_at_write_read_premium(self): + # 1M re-written tokens, all previously cached → full premium. + wasted = compute_wasted_usd( + CacheStatus.MISS_AVOIDABLE, + cache_write_tokens=1_000_000, + previous_cached_prefix_tokens=2_000_000, + pricing_snapshot=PRICING, + ) + assert wasted == (3.75 - 0.30) + + def test_rewritten_capped_at_previous_cached_prefix(self): + # Only 100k of the 1M written were cached before — the new suffix + # would have been written under a hit too, so it isn't waste. + wasted = compute_wasted_usd( + CacheStatus.MISS_AVOIDABLE, + cache_write_tokens=1_000_000, + previous_cached_prefix_tokens=100_000, + pricing_snapshot=PRICING, + ) + assert wasted == (100_000 / 1_000_000) * (3.75 - 0.30) + + def test_unknown_previous_prefix_uses_full_write(self): + wasted = compute_wasted_usd( + CacheStatus.MISS_AVOIDABLE, + cache_write_tokens=500_000, + previous_cached_prefix_tokens=None, + pricing_snapshot=PRICING, + ) + assert wasted == (500_000 / 1_000_000) * (3.75 - 0.30) + + def test_zero_for_non_avoidable_statuses(self): + for status in ( + CacheStatus.FIRST_WRITE, + CacheStatus.HIT, + CacheStatus.MISS_TTL_EXPIRED, + CacheStatus.UNCACHED, + ): + assert ( + compute_wasted_usd(status, 1_000_000, 1_000_000, PRICING) == 0.0 + ) + + def test_zero_without_pricing_or_with_none_prices(self): + assert compute_wasted_usd(CacheStatus.MISS_AVOIDABLE, 1000, 1000, None) == 0.0 + # Rows can store explicit None prices (managed models). + assert ( + compute_wasted_usd( + CacheStatus.MISS_AVOIDABLE, + 1000, + 1000, + {"cacheWritePricePerMtok": None, "cacheReadPricePerMtok": None}, + ) + == 0.0 + ) + + +class TestFingerprints: + def test_text_fingerprint_stable_and_short(self): + assert fingerprint_text("hello") == fingerprint_text("hello") + assert fingerprint_text("hello") != fingerprint_text("hello!") + assert len(fingerprint_text("hello")) == 16 + assert fingerprint_text(None) == fingerprint_text("") + + def test_dict_key_order_does_not_matter(self): + a = {"b": 1, "a": [1, 2]} + b = {"a": [1, 2], "b": 1} + assert fingerprint_canonical_json(a) == fingerprint_canonical_json(b) + + def test_list_order_matters(self): + # Deliberate: tool-spec / message order is what Bedrock's + # exact-prefix match is sensitive to. + assert fingerprint_canonical_json([1, 2]) != fingerprint_canonical_json([2, 1]) + + def test_non_json_values_do_not_raise(self): + assert isinstance(fingerprint_canonical_json({"x": object()}), str) + + +class TestEmfEmission: + def _capture(self, **kwargs): + from apis.shared.observability import emf + + stream = io.StringIO() + handler = logging.StreamHandler(stream) + handler.setFormatter(logging.Formatter("%(message)s")) + emf._emf_logger.addHandler(handler) + try: + emit_prompt_cache_metrics(**kwargs) + finally: + emf._emf_logger.removeHandler(handler) + return stream.getvalue().strip() + + def test_record_is_raw_json_with_emf_directive(self): + line = self._capture( + cache_read_tokens=1000, + cache_write_tokens=200, + avoidable_miss=True, + wasted_usd=0.0123456789, + model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0", + session_id="sess-1", + cache_status="miss_avoidable", + ) + record = json.loads(line) # the whole line must be the JSON object + directive = record["_aws"]["CloudWatchMetrics"][0] + metric_names = {m["Name"] for m in directive["Metrics"]} + assert metric_names == { + "CacheReadTokens", + "CacheWriteTokens", + "AvoidableMiss", + "WastedUsd", + } + assert record["CacheReadTokens"] == 1000 + assert record["CacheWriteTokens"] == 200 + assert record["AvoidableMiss"] == 1 + assert record["WastedUsd"] == 0.012346 # rounded to 6 places + assert record["modelId"].startswith("us.anthropic") + assert record["cacheStatus"] == "miss_avoidable" + + def test_no_miss_and_defaults(self): + line = self._capture( + cache_read_tokens=0, cache_write_tokens=0, avoidable_miss=False + ) + record = json.loads(line) + assert record["AvoidableMiss"] == 0 + assert record["WastedUsd"] == 0.0 + assert "modelId" not in record diff --git a/backend/tests/shared/test_skills_repository.py b/backend/tests/shared/test_skills_repository.py index 9c156b5fe..3e9a1e5d9 100644 --- a/backend/tests/shared/test_skills_repository.py +++ b/backend/tests/shared/test_skills_repository.py @@ -169,6 +169,31 @@ async def test_batch_get_skills(self, skill_repository): skills = await skill_repository.batch_get_skills(["skill_one", "skill_two", "ghost"]) assert {s.skill_id for s in skills} == {"skill_one", "skill_two"} + @pytest.mark.asyncio + async def test_batch_get_skills_sorted_regardless_of_response_order( + self, skill_repository, monkeypatch + ): + # DynamoDB batch_get_item response order is arbitrary; these records + # feed the system-prompt block, where an order flip + # between turns invalidates the Bedrock prompt cache. Force a + # descending response order and assert the repo still returns sorted. + for sid in ("zeta", "alpha", "midway"): + await skill_repository.create_skill(_make_skill(sid)) + + client = skill_repository._dynamodb.meta.client + real_batch_get = client.batch_get_item + + def descending_batch_get(**kwargs): + response = real_batch_get(**kwargs) + for items in response.get("Responses", {}).values(): + items.sort(key=lambda item: item["PK"], reverse=True) + return response + + monkeypatch.setattr(client, "batch_get_item", descending_batch_get) + + skills = await skill_repository.batch_get_skills(["zeta", "alpha", "midway"]) + assert [s.skill_id for s in skills] == ["alpha", "midway", "zeta"] + @pytest.mark.asyncio async def test_batch_get_empty(self, skill_repository): assert await skill_repository.batch_get_skills([]) == [] diff --git a/backend/uv.lock b/backend/uv.lock index 1c575a2fc..48a5cada1 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -12,7 +12,7 @@ resolution-markers = [ [[package]] name = "agentcore-stack" -version = "1.8.0" +version = "1.9.0" source = { editable = "." } dependencies = [ { name = "aiofiles" }, diff --git a/docs/kaizen/review-queue.md b/docs/kaizen/review-queue.md index 5f7eeecc9..c1dff5f71 100644 --- a/docs/kaizen/review-queue.md +++ b/docs/kaizen/review-queue.md @@ -5,6 +5,20 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Open +### [2026-07-19] Track harness-sdk#3348 (rolling pair of message cachePoints) — local workaround gated on dashboard evidence +- **Source**: Phil-initiated (PR #697 follow-up) — https://github.com/strands-agents/harness-sdk/issues/3348 (filed by philmerrell, open, no maintainer response yet); prod session aecd387d (18-way parallel tool fan-out → cacheRead=0 / cacheWrite=134k mid-turn, the ~20-block Anthropic lookback miss mode documented in `model_config.py:366`) +- **Surface**: backend (`agents/main_agent/core/model_config.py` 3-cachePoint budget). If built locally: strands 1.48's `_inject_cache_point` **strips any pre-existing message-level cachePoints**, so a rolling pair requires dropping `CacheConfig(strategy="auto")` and hand-placing both points via a hook — the 4th Bedrock cachePoint slot is free. Position tests in `tests/agents/main_agent/core/test_bedrock_cache_points.py` are the safety net. +- **Effort × Impact**: (track) free × M; (local workaround) M × M — but high regression risk in the highest cost-of-regression area +- **Subtracts**: no — upstream-native fix preferred; a local workaround would later be deleted in favor of it +- **Status**: open — check #3348 each scan. Do NOT build the local workaround until the `AvoidableMiss`/`WastedUsd` dashboard (PR #697 follow-up, in flight) shows the fan-out miss mode fires often enough to earn the risk. + +### [2026-07-19] Spike: ContextOffloader adoption (ingestion-time tool-result offload → S3) +- **Source**: Phil-initiated (PR #697 follow-up) — strands 1.48.0 ships `ContextOffloader` as a vended plugin (`strands/vended_plugins/context_offloader/`): hooks `AfterToolCallEvent`, token-gates results (default 2,500 via `model.count_tokens`), stores oversized blocks, rewrites to preview + reference, registers `retrieve_offloaded_content`. Relates to long-open issue #266 (large tool-result offload). +- **Surface**: backend (`agent_factory.py` — `plugins=` already plumbed; a custom S3 `Storage` backend is the real build — the plugin ships InMemory/File only, and references must survive AgentCore Runtime restores across turns) +- **Effort × Impact**: M–H × M–H (payoff directly measurable by the PR #697 cache/cost metrics) +- **Subtracts**: partial — bounds MCP/tool payload growth at the source instead of relying solely on reactive below-anchor truncation in `TurnBasedSessionManager` (which stays for legacy history) +- **Status**: open — spike before commitment; four known gotchas: (1) `evict_after_cycles=20` runs on `BeforeModelCallEvent` and touches *prior* messages — potential byte-stability cache-buster, verify semantics or set `None` and expire via S3 lifecycle; (2) `model.count_tokens` per tool result adds latency, and Bedrock CountTokens rejects `us.*` inference-profile ids (de-prefix precedent in context attribution); (3) adoption flips `toolConfigHash` once (expected; the new tool must land in the deterministically-ordered tool list); (4) check SPA tool-result rendering against placeholder content. + ### [2026-07-17] Bump `bedrock-agentcore` 1.9.1 → 1.18.0 (closes #571 cross-process Memory reorder + #482 SSE deadlock) - **Source**: research/2026-07-17.md ▸ Top 5 #1 — bedrock-agentcore 1.18.0 (Jul 10, PR #572/#573 close #571; #482 fix in 1.17.0/PR #563) — https://github.com/aws/bedrock-agentcore-sdk-python/releases - **Surface**: backend (`backend/pyproject.toml` + coupled `boto3`; inference-api chat router + `TurnBasedSessionManager` flush ordering; full local pytest suite) diff --git a/frontend/ai.client/package-lock.json b/frontend/ai.client/package-lock.json index 92fcfcfe6..0e6b860e5 100644 --- a/frontend/ai.client/package-lock.json +++ b/frontend/ai.client/package-lock.json @@ -1,12 +1,12 @@ { "name": "ai.client", - "version": "1.8.0", + "version": "1.9.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "ai.client", - "version": "1.8.0", + "version": "1.9.0", "dependencies": { "@angular/cdk": "21.2.14", "@angular/common": "21.2.17", diff --git a/frontend/ai.client/package.json b/frontend/ai.client/package.json index 359bc4d20..051419903 100644 --- a/frontend/ai.client/package.json +++ b/frontend/ai.client/package.json @@ -1,6 +1,6 @@ { "name": "ai.client", - "version": "1.8.0", + "version": "1.9.0", "scripts": { "ng": "ng", "start": "ng serve", diff --git a/frontend/ai.client/src/app/admin/admin.routes.ts b/frontend/ai.client/src/app/admin/admin.routes.ts index 8b95ca798..1596493d4 100644 --- a/frontend/ai.client/src/app/admin/admin.routes.ts +++ b/frontend/ai.client/src/app/admin/admin.routes.ts @@ -11,6 +11,10 @@ export const adminRoutes: Routes = [ path: 'costs', loadComponent: () => import('./costs/admin-costs.page').then(m => m.AdminCostsPage), }, + { + path: 'costs/sessions/:id', + loadComponent: () => import('./costs/pages/session-cost-anatomy.page').then(m => m.SessionCostAnatomyPage), + }, { path: 'quota', loadChildren: () => import('./quota-tiers/quota-routing.module').then(m => m.quotaRoutes), diff --git a/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts b/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts index 37acd5c97..3dc31f88d 100644 --- a/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts +++ b/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts @@ -7,10 +7,12 @@ import { signal, } from '@angular/core'; import { Router } from '@angular/router'; +import { FormsModule } from '@angular/forms'; import { NgIcon, provideIcons } from '@ng-icons/core'; import { heroArrowLeft, heroArrowDownTray, + heroMagnifyingGlass, } from '@ng-icons/heroicons/outline'; import { AdminCostStateService } from './services'; import { PeriodSelectorComponent } from './components/period-selector.component'; @@ -28,6 +30,7 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component' @Component({ selector: 'app-admin-costs', imports: [ + FormsModule, NgIcon, PeriodSelectorComponent, SystemSummaryCardComponent, @@ -35,7 +38,7 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component' CostTrendsChartComponent, ModelBreakdownComponent, ], - providers: [provideIcons({ heroArrowLeft, heroArrowDownTray })], + providers: [provideIcons({ heroArrowLeft, heroArrowDownTray, heroMagnifyingGlass })], changeDetection: ChangeDetectionStrategy.OnPush, template: `
@@ -66,6 +69,46 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component'
+ +
+
+ +

+ Look up per-call cache diagnostics for a session ID. +

+
+
+
+
+ +
+
+ @if (loading()) {
@@ -177,6 +220,9 @@ export class AdminCostsPage implements OnInit { trends = this.stateService.trends; modelUsage = this.stateService.modelUsage; + // Session-id lookup for the cost-anatomy drill-down + sessionLookupId = signal(''); + // Track pagination state for top users private topUsersLimit = signal(20); hasMoreUsers = computed( @@ -238,6 +284,13 @@ export class AdminCostsPage implements OnInit { this.router.navigate(['/admin/users', userId]); } + onInspectSession(): void { + const sessionId = this.sessionLookupId().trim(); + if (sessionId) { + this.router.navigate(['/admin/costs/sessions', sessionId]); + } + } + async onLoadMoreUsers(): Promise { const newLimit = this.topUsersLimit() + 20; this.topUsersLimit.set(newLimit); diff --git a/frontend/ai.client/src/app/admin/costs/models/admin-cost.models.ts b/frontend/ai.client/src/app/admin/costs/models/admin-cost.models.ts index 25935db24..677654228 100644 --- a/frontend/ai.client/src/app/admin/costs/models/admin-cost.models.ts +++ b/frontend/ai.client/src/app/admin/costs/models/admin-cost.models.ts @@ -91,6 +91,59 @@ export interface AdminCostDashboard { dailyTrends?: CostTrend[]; } +// ========== Session Cost Anatomy ========== + +/** + * Derived prompt-cache status for one model call. + * Null on rows persisted before the cache-observability feature. + */ +export type CacheStatus = + | 'first_write' + | 'hit' + | 'miss_ttl_expired' + | 'miss_avoidable' + | 'uncached'; + +/** Prompt-cache prefix hashes for one model call. */ +export interface PrefixFingerprints { + toolConfigHash?: string | null; + systemPromptHash?: string | null; + historyHash?: string | null; + messageCount?: number | null; +} + +/** One model call within a session's cost anatomy. */ +export interface SessionCallRow { + timestamp: string; + messageId?: number | null; + modelId?: string | null; + + inputTokens: number; + outputTokens: number; + cacheReadTokens: number; + cacheWriteTokens: number; + + cost: number; + cacheStatus?: CacheStatus | null; + cacheGapSeconds?: number | null; + wastedUsd: number; + prefixFingerprints?: PrefixFingerprints | null; +} + +/** Per-call cost anatomy for one session (admin cache-miss forensics). */ +export interface SessionCostAnatomy { + sessionId: string; + calls: SessionCallRow[]; + + totalCost: number; + totalCacheReadTokens: number; + totalCacheWriteTokens: number; + avoidableMissCount: number; + wastedUsd: number; + /** cacheRead / (cacheRead + cacheWrite); null until any cache activity. */ + cacheEfficiency: number | null; +} + // ========== API Request Options ========== export interface DashboardRequestOptions { diff --git a/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.spec.ts b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.spec.ts new file mode 100644 index 000000000..836485687 --- /dev/null +++ b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.spec.ts @@ -0,0 +1,160 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { TestBed } from '@angular/core/testing'; +import { provideRouter } from '@angular/router'; +import { HttpErrorResponse } from '@angular/common/http'; +import { of, throwError } from 'rxjs'; +import { SessionCostAnatomyPage } from './session-cost-anatomy.page'; +import { AdminCostHttpService } from '../services/admin-cost-http.service'; +import { SessionCostAnatomy } from '../models'; + +const MOCK_ANATOMY: SessionCostAnatomy = { + sessionId: 'sess-1', + totalCost: 0.42, + totalCacheReadTokens: 20_000, + totalCacheWriteTokens: 5_000, + avoidableMissCount: 1, + wastedUsd: 0.03, + cacheEfficiency: 0.8, + calls: [ + { + timestamp: '2026-07-19T10:00:00Z', + messageId: 1, + modelId: 'us.anthropic.claude-sonnet-5', + inputTokens: 100, + outputTokens: 50, + cacheReadTokens: 0, + cacheWriteTokens: 5_000, + cost: 0.1, + cacheStatus: 'first_write', + cacheGapSeconds: null, + wastedUsd: 0, + prefixFingerprints: { toolConfigHash: 'aaaa1111', systemPromptHash: 'bbbb2222', historyHash: 'cccc3333', messageCount: 2 }, + }, + { + timestamp: '2026-07-19T10:01:00Z', + messageId: 2, + modelId: 'us.anthropic.claude-sonnet-5', + inputTokens: 120, + outputTokens: 60, + cacheReadTokens: 0, + cacheWriteTokens: 5_200, + cost: 0.12, + cacheStatus: 'miss_avoidable', + cacheGapSeconds: 60, + wastedUsd: 0.03, + prefixFingerprints: { toolConfigHash: 'DIFFERENT', systemPromptHash: 'bbbb2222', historyHash: 'cccc3333', messageCount: 4 }, + }, + ], +}; + +describe('SessionCostAnatomyPage', () => { + let getSessionCostAnatomy: ReturnType; + + function setup(mock: ReturnType) { + getSessionCostAnatomy = mock; + TestBed.resetTestingModule(); + TestBed.configureTestingModule({ + providers: [ + provideRouter([]), + { provide: AdminCostHttpService, useValue: { getSessionCostAnatomy } }, + ], + }); + TestBed.overrideComponent(SessionCostAnatomyPage, { + set: { template: '
' }, + }); + const fixture = TestBed.createComponent(SessionCostAnatomyPage); + fixture.componentRef.setInput('id', 'sess-1'); + fixture.detectChanges(); + return fixture; + } + + afterEach(() => { + TestBed.resetTestingModule(); + }); + + it('loads the anatomy for the routed session id and annotates fingerprint diffs', async () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + await vi.waitFor(() => { + expect(page.anatomyResource.hasValue()).toBe(true); + }); + + expect(getSessionCostAnatomy).toHaveBeenCalledWith('sess-1'); + const rows = page.rows(); + expect(rows).toHaveLength(2); + expect(rows[0].changed).toEqual([]); + // The flipped toolConfigHash is the diagnosis on the miss_avoidable row. + expect(rows[1].changed).toEqual(['toolConfigHash']); + expect(page.notFound()).toBe(false); + }); + + it('treats a 404 as "session has no cost rows"', async () => { + const fixture = setup( + vi.fn().mockReturnValue( + throwError(() => new HttpErrorResponse({ status: 404, statusText: 'Not Found' })) + ) + ); + const page = fixture.componentInstance; + + await vi.waitFor(() => { + expect(page.anatomyResource.error()).toBeTruthy(); + }); + expect(page.notFound()).toBe(true); + }); + + it('does not report notFound for other errors', async () => { + const fixture = setup( + vi.fn().mockReturnValue( + throwError(() => new HttpErrorResponse({ status: 500, statusText: 'Server Error' })) + ) + ); + const page = fixture.componentInstance; + + await vi.waitFor(() => { + expect(page.anatomyResource.error()).toBeTruthy(); + }); + expect(page.notFound()).toBe(false); + }); + + it('toggles row expansion', () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + expect(page.isExpanded(0)).toBe(false); + page.toggleExpand(0); + expect(page.isExpanded(0)).toBe(true); + page.toggleExpand(0); + expect(page.isExpanded(0)).toBe(false); + }); + + it('formats cache efficiency, handling null', () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + expect(page.formatEfficiency(null)).toBe('—'); + expect(page.formatEfficiency(0.8)).toBe('80.0%'); + }); + + it('formats cache gaps, handling null', () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + expect(page.formatGap(null)).toBe('—'); + expect(page.formatGap(undefined)).toBe('—'); + expect(page.formatGap(42)).toBe('42s'); + expect(page.formatGap(60)).toBe('1m'); + expect(page.formatGap(312)).toBe('5m 12s'); + }); + + it('maps cache statuses to color-coded badge classes', () => { + const fixture = setup(vi.fn().mockReturnValue(of(MOCK_ANATOMY))); + const page = fixture.componentInstance; + + expect(page.getStatusClass('hit')).toContain('bg-green-100'); + expect(page.getStatusClass('first_write')).toContain('bg-blue-100'); + expect(page.getStatusClass('miss_ttl_expired')).toContain('bg-yellow-100'); + expect(page.getStatusClass('miss_avoidable')).toContain('bg-red-100'); + expect(page.getStatusClass('uncached')).toContain('bg-gray-100'); + }); +}); diff --git a/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.ts b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.ts new file mode 100644 index 000000000..96197b730 --- /dev/null +++ b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.page.ts @@ -0,0 +1,469 @@ +import { + ChangeDetectionStrategy, + Component, + computed, + inject, + input, + resource, + signal, +} from '@angular/core'; +import { DatePipe } from '@angular/common'; +import { HttpErrorResponse } from '@angular/common/http'; +import { RouterLink } from '@angular/router'; +import { firstValueFrom } from 'rxjs'; +import { NgIcon, provideIcons } from '@ng-icons/core'; +import { heroArrowLeft, heroChevronDown } from '@ng-icons/heroicons/outline'; +import { AdminCostHttpService } from '../services/admin-cost-http.service'; +import { CacheStatus, SessionCallRow } from '../models'; +import { + AnatomyRow, + FINGERPRINT_KEYS, + FINGERPRINT_LABELS, + FingerprintKey, + buildAnatomyRows, + truncateHash, +} from './session-cost-anatomy.util'; + +/** + * Admin drill-down: per-model-call cost anatomy for one session. + * + * The forensic view for prompt-cache diagnostics — each call row carries its + * cache status and prefix-fingerprint hashes, and the hash that flipped + * between consecutive calls names the cache-buster (the diagnosis on + * `miss_avoidable` rows). + */ +@Component({ + selector: 'app-session-cost-anatomy', + imports: [RouterLink, NgIcon, DatePipe], + providers: [provideIcons({ heroArrowLeft, heroChevronDown })], + changeDetection: ChangeDetectionStrategy.OnPush, + template: ` +
+ + + + + +
+

Session Cost Anatomy

+

+ {{ id() }} +

+
+ + @if (anatomyResource.isLoading()) { + +
+
+
+

Loading session cost anatomy…

+
+
+ } @else if (notFound()) { + +
+

No cost rows for this session

+

+ Nothing has been recorded under this session ID — check the ID, or the session may predate + cost tracking. +

+
+ } @else if (anatomyResource.error()) { + +
+

Failed to load session cost anatomy. Please try again.

+ +
+ } @else if (anatomyResource.hasValue()) { + +
+
+

+ Total Cost +

+

+ {{ formatCurrency(anatomyResource.value().totalCost) }} +

+
+
+

+ Cache Efficiency +

+

+ {{ formatEfficiency(anatomyResource.value().cacheEfficiency) }} +

+
+
+

+ Avoidable Misses +

+

+ {{ anatomyResource.value().avoidableMissCount }} +

+
+
+

+ Wasted +

+

+ {{ formatCurrency(anatomyResource.value().wastedUsd, 4) }} +

+
+
+

+ Cache Read +

+

+ {{ formatTokens(anatomyResource.value().totalCacheReadTokens) }} +

+
+
+

+ Cache Write +

+

+ {{ formatTokens(anatomyResource.value().totalCacheWriteTokens) }} +

+
+
+ + +
+
+ + + + + + + + + + + + + + + + + + + @for (row of rows(); track row.index) { + + + + + + + + + + + + + + + @if (isExpanded(row.index)) { + + + + } + } @empty { + + + + } + +
ExpandTimeModelInReadWriteOutCostCache StatusGapWastedFingerprints
+ + + {{ row.call.timestamp | date: 'MMM d, HH:mm:ss' }} + + {{ row.call.modelId ?? '—' }} + + {{ formatTokens(row.call.inputTokens) }} + + {{ formatTokens(row.call.cacheReadTokens) }} + + {{ formatTokens(row.call.cacheWriteTokens) }} + + {{ formatTokens(row.call.outputTokens) }} + + {{ formatCurrency(row.call.cost, 4) }} + + @if (row.call.cacheStatus; as status) { + {{ getStatusLabel(status) }} + } @else { + — + } + + {{ formatGap(row.call.cacheGapSeconds) }} + + {{ row.call.wastedUsd > 0 ? formatCurrency(row.call.wastedUsd, 4) : '—' }} + + @if (row.call.prefixFingerprints; as fp) { +
+ @for (key of fingerprintKeys; track key) { + + {{ fingerprintLabels[key] }} {{ truncateHash(fp[key]) }} + + } +
+ } @else { + — + } +
+
+
+
+ Timestamp +
+
+ {{ row.call.timestamp }} +
+
+
+
+ Message ID +
+
+ {{ row.call.messageId ?? '—' }} +
+
+
+
+ Message Count +
+
+ {{ row.call.prefixFingerprints?.messageCount ?? '—' }} +
+
+ @for (key of fingerprintKeys; track key) { +
+
+ {{ fingerprintLabels[key] }} Hash + @if (isChanged(row, key)) { + (changed) + } +
+
+ {{ row.call.prefixFingerprints?.[key] ?? '—' }} +
+
+ } +
+
+ No model calls recorded for this session. +
+
+
+ } +
+ `, +}) +export class SessionCostAnatomyPage { + private costHttp = inject(AdminCostHttpService); + + /** Session ID from the `costs/sessions/:id` route (component input binding). */ + readonly id = input.required(); + + readonly fingerprintKeys = FINGERPRINT_KEYS; + readonly fingerprintLabels = FINGERPRINT_LABELS; + readonly truncateHash = truncateHash; + + readonly anatomyResource = resource({ + params: () => ({ id: this.id() }), + loader: ({ params }) => firstValueFrom(this.costHttp.getSessionCostAnatomy(params.id)), + }); + + /** Chronological call rows annotated with fingerprint diffs. */ + readonly rows = computed(() => + this.anatomyResource.hasValue() ? buildAnatomyRows(this.anatomyResource.value().calls) : [] + ); + + /** True when the backend returned 404 — the session has no cost rows. */ + readonly notFound = computed(() => this.errorStatus(this.anatomyResource.error()) === 404); + + private expandedRows = signal>(new Set()); + + isExpanded(index: number): boolean { + return this.expandedRows().has(index); + } + + toggleExpand(index: number): void { + this.expandedRows.update((current) => { + const next = new Set(current); + if (next.has(index)) { + next.delete(index); + } else { + next.add(index); + } + return next; + }); + } + + isChanged(row: AnatomyRow, key: FingerprintKey): boolean { + return row.changed.includes(key); + } + + getFingerprintClass(row: AnatomyRow, key: FingerprintKey): string { + const base = 'inline-flex items-center rounded-2xl px-2 py-0.5 font-mono text-xs/5'; + if (this.isChanged(row, key)) { + return `${base} bg-red-100 font-semibold text-red-700 ring-1 ring-inset ring-red-300 dark:bg-red-900/40 dark:text-red-300 dark:ring-red-700`; + } + return `${base} bg-gray-100 text-gray-600 dark:bg-gray-700 dark:text-gray-400`; + } + + getStatusClass(status: CacheStatus): string { + const base = 'inline-flex items-center rounded-2xl px-2.5 py-0.5 text-xs/5 font-medium'; + switch (status) { + case 'hit': + return `${base} bg-green-100 text-green-800 dark:bg-green-900/30 dark:text-green-300`; + case 'first_write': + return `${base} bg-blue-100 text-blue-800 dark:bg-blue-900/30 dark:text-blue-300`; + case 'miss_ttl_expired': + return `${base} bg-yellow-100 text-yellow-800 dark:bg-yellow-900/30 dark:text-yellow-300`; + case 'miss_avoidable': + return `${base} bg-red-100 text-red-800 dark:bg-red-900/30 dark:text-red-300`; + case 'uncached': + default: + return `${base} bg-gray-100 text-gray-800 dark:bg-gray-700 dark:text-gray-300`; + } + } + + getStatusLabel(status: CacheStatus): string { + switch (status) { + case 'hit': + return 'Hit'; + case 'first_write': + return 'First Write'; + case 'miss_ttl_expired': + return 'Miss (TTL)'; + case 'miss_avoidable': + return 'Miss (Avoidable)'; + case 'uncached': + return 'Uncached'; + default: + return status; + } + } + + formatCurrency(value: number, maxFractionDigits = 2): string { + return new Intl.NumberFormat('en-US', { + style: 'currency', + currency: 'USD', + minimumFractionDigits: 2, + maximumFractionDigits: maxFractionDigits, + }).format(value); + } + + formatTokens(tokens: number): string { + if (tokens >= 1_000_000) { + return `${(tokens / 1_000_000).toFixed(1)}M`; + } else if (tokens >= 1_000) { + return `${(tokens / 1_000).toFixed(1)}K`; + } + return tokens.toString(); + } + + formatEfficiency(efficiency: number | null): string { + if (efficiency === null) { + return '—'; + } + return `${(efficiency * 100).toFixed(1)}%`; + } + + formatGap(seconds: number | null | undefined): string { + if (seconds == null) { + return '—'; + } + if (seconds >= 60) { + const minutes = Math.floor(seconds / 60); + const rest = seconds % 60; + return rest > 0 ? `${minutes}m ${rest}s` : `${minutes}m`; + } + return `${seconds}s`; + } + + private errorStatus(error: unknown): number | undefined { + if (error instanceof HttpErrorResponse) { + return error.status; + } + // resource() may wrap loader errors; check the cause chain. + const cause = (error as { cause?: unknown } | null | undefined)?.cause; + if (cause instanceof HttpErrorResponse) { + return cause.status; + } + return undefined; + } +} diff --git a/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.spec.ts b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.spec.ts new file mode 100644 index 000000000..8d77e6373 --- /dev/null +++ b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.spec.ts @@ -0,0 +1,98 @@ +import { describe, it, expect } from 'vitest'; +import { buildAnatomyRows, truncateHash } from './session-cost-anatomy.util'; +import { SessionCallRow } from '../models'; + +function makeCall(overrides: Partial = {}): SessionCallRow { + return { + timestamp: '2026-07-19T10:00:00Z', + inputTokens: 100, + outputTokens: 50, + cacheReadTokens: 0, + cacheWriteTokens: 0, + cost: 0.01, + wastedUsd: 0, + ...overrides, + }; +} + +describe('buildAnatomyRows', () => { + it('never flags the first call', () => { + const rows = buildAnatomyRows([ + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + ]); + expect(rows).toHaveLength(1); + expect(rows[0].changed).toEqual([]); + }); + + it('flags exactly the hash that flipped between consecutive calls', () => { + const rows = buildAnatomyRows([ + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + makeCall({ + prefixFingerprints: { toolConfigHash: 'XXX', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + ]); + expect(rows[1].changed).toEqual(['toolConfigHash']); + }); + + it('flags multiple flipped hashes', () => { + const rows = buildAnatomyRows([ + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'YYY', historyHash: 'ZZZ' }, + }), + ]); + expect(rows[1].changed).toEqual(['systemPromptHash', 'historyHash']); + }); + + it('skips rows without fingerprints as comparison baselines', () => { + const rows = buildAnatomyRows([ + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + makeCall({ prefixFingerprints: null }), + makeCall({ + prefixFingerprints: { toolConfigHash: 'XXX', systemPromptHash: 'bbb', historyHash: 'ccc' }, + }), + ]); + // Row 1 has nothing to diff; row 2 diffs against row 0, not the null row. + expect(rows[1].changed).toEqual([]); + expect(rows[2].changed).toEqual(['toolConfigHash']); + }); + + it('does not flag a hash when either side is missing', () => { + const rows = buildAnatomyRows([ + makeCall({ prefixFingerprints: { toolConfigHash: 'aaa', historyHash: 'ccc' } }), + makeCall({ + prefixFingerprints: { toolConfigHash: 'aaa', systemPromptHash: 'now-present', historyHash: 'ccc' }, + }), + ]); + expect(rows[1].changed).toEqual([]); + }); + + it('does not flag identical fingerprints', () => { + const fp = { toolConfigHash: 'aaa', systemPromptHash: 'bbb', historyHash: 'ccc' }; + const rows = buildAnatomyRows([ + makeCall({ prefixFingerprints: fp }), + makeCall({ prefixFingerprints: { ...fp } }), + ]); + expect(rows[1].changed).toEqual([]); + }); +}); + +describe('truncateHash', () => { + it('truncates to the first 8 chars', () => { + expect(truncateHash('0123456789abcdef')).toBe('01234567'); + }); + + it('returns an em dash for missing hashes', () => { + expect(truncateHash(null)).toBe('—'); + expect(truncateHash(undefined)).toBe('—'); + expect(truncateHash('')).toBe('—'); + }); +}); diff --git a/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.ts b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.ts new file mode 100644 index 000000000..76907398a --- /dev/null +++ b/frontend/ai.client/src/app/admin/costs/pages/session-cost-anatomy.util.ts @@ -0,0 +1,62 @@ +import { PrefixFingerprints, SessionCallRow } from '../models'; + +export type FingerprintKey = 'toolConfigHash' | 'systemPromptHash' | 'historyHash'; + +export const FINGERPRINT_KEYS: readonly FingerprintKey[] = [ + 'toolConfigHash', + 'systemPromptHash', + 'historyHash', +]; + +export const FINGERPRINT_LABELS: Record = { + toolConfigHash: 'Tools', + systemPromptHash: 'System', + historyHash: 'History', +}; + +/** One call row annotated with fingerprint-diff results against the previous fingerprinted call. */ +export interface AnatomyRow { + call: SessionCallRow; + index: number; + /** Fingerprint hashes that flipped vs. the nearest previous call that has fingerprints. */ + changed: FingerprintKey[]; +} + +/** + * Annotate chronological call rows with which prefix-fingerprint hashes + * changed since the previous fingerprinted call — on a `miss_avoidable` + * row, the flipped hash names the cache-buster. + * + * A hash is flagged only when both rows carry a value for it and the values + * differ; rows predating the fingerprint feature (null fingerprints) are + * skipped as comparison baselines. + */ +export function buildAnatomyRows(calls: SessionCallRow[]): AnatomyRow[] { + let previous: PrefixFingerprints | undefined; + + return calls.map((call, index) => { + const fingerprints = call.prefixFingerprints ?? undefined; + const changed: FingerprintKey[] = []; + + if (fingerprints && previous) { + for (const key of FINGERPRINT_KEYS) { + const current = fingerprints[key]; + const before = previous[key]; + if (current != null && before != null && current !== before) { + changed.push(key); + } + } + } + + if (fingerprints) { + previous = fingerprints; + } + + return { call, index, changed }; + }); +} + +/** First 8 chars of a fingerprint hash, or an em dash when absent. */ +export function truncateHash(hash: string | null | undefined): string { + return hash ? hash.slice(0, 8) : '—'; +} diff --git a/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.spec.ts b/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.spec.ts index 67f5a9e09..82ad514d2 100644 --- a/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.spec.ts +++ b/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.spec.ts @@ -65,6 +65,35 @@ describe('AdminCostHttpService', () => { req.flush(mockSummary); }); + it('should get session cost anatomy', () => { + const mockAnatomy = { + sessionId: 'sess-1', + calls: [], + totalCost: 1.23, + totalCacheReadTokens: 1000, + totalCacheWriteTokens: 500, + avoidableMissCount: 0, + wastedUsd: 0, + cacheEfficiency: 2 / 3, + }; + + service.getSessionCostAnatomy('sess-1').subscribe(anatomy => { + expect(anatomy).toEqual(mockAnatomy); + }); + + const req = httpMock.expectOne('http://localhost:8000/admin/costs/sessions/sess-1/calls'); + expect(req.request.method).toBe('GET'); + req.flush(mockAnatomy); + }); + + it('should URL-encode the session id in the anatomy request', () => { + service.getSessionCostAnatomy('a/b c').subscribe(); + + const req = httpMock.expectOne('http://localhost:8000/admin/costs/sessions/a%2Fb%20c/calls'); + expect(req.request.method).toBe('GET'); + req.flush({ sessionId: 'a/b c', calls: [] }); + }); + it('should export data', () => { const mockBlob = new Blob(['csv data'], { type: 'text/csv' }); diff --git a/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.ts b/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.ts index 5f79bac74..b83793ffe 100644 --- a/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.ts +++ b/frontend/ai.client/src/app/admin/costs/services/admin-cost-http.service.ts @@ -12,6 +12,7 @@ import { DashboardRequestOptions, TopUsersRequestOptions, TrendsRequestOptions, + SessionCostAnatomy, } from '../models'; /** @@ -128,6 +129,17 @@ export class AdminCostHttpService { return this.http.get(`${this.baseUrl()}/trends`, { params }); } + /** + * Get the per-model-call cost anatomy for one session. + * Returns chronological call rows with cache status, prefix fingerprints, + * and session-level cache rollups. 404 when the session has no cost rows. + */ + getSessionCostAnatomy(sessionId: string): Observable { + return this.http.get( + `${this.baseUrl()}/sessions/${encodeURIComponent(sessionId)}/calls` + ); + } + /** * Export cost data for a period. * Returns blob for download. diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css index 2bcb6cfb9..e0ccd0cfb 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css @@ -6,26 +6,4 @@ transition: border-color 0.2s ease-in-out, box-shadow 0.2s ease-in-out; } -.chat-textarea { - min-height: 3rem; - max-height: 12rem; - transition: border-color 0.2s ease-in-out, box-shadow 0.2s ease-in-out; - font-family: inherit; - font-size: 0.875rem; - line-height: 1.5rem; -} - -/* Auto-resize textarea */ -.chat-textarea { - field-sizing: content; -} - -@supports not (field-sizing: content) { - .chat-textarea { - height: auto; - } -} - - - diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html index 6428a4681..1e0193146 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html @@ -60,7 +60,7 @@ (focus)="onFocus()" (blur)="onBlur()" placeholder="How can I help you today?" - class="block w-full resize-none bg-transparent px-6 py-4 text-base text-gray-900 placeholder:text-gray-500 dark:text-gray-100 dark:placeholder:text-gray-400 focus:outline-hidden overflow-hidden" + class="block w-full resize-none bg-transparent px-6 py-4 text-base text-gray-900 placeholder:text-gray-500 dark:text-gray-100 dark:placeholder:text-gray-400 focus:outline-hidden overflow-y-auto" style="min-height: 60px; max-height: 200px;" > diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts index 91853a6d7..010f3beed 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts @@ -38,6 +38,11 @@ import { ToolService } from '../../../services/tool/tool.service'; import { VoiceChatService, type VoiceStatus } from '../../services/voice'; import { SystemPromptsService } from '../../../services/system-prompts/system-prompts.service'; +// Must stay in sync with the inline min-height/max-height on the textarea in +// chat-input.component.html. +const MIN_TEXTAREA_HEIGHT_PX = 60; +const MAX_TEXTAREA_HEIGHT_PX = 200; + interface Message { content: string; timestamp: Date; @@ -97,7 +102,6 @@ export class ChatInputComponent { // Signals for state management userInput = signal(''); - isExpanded = signal(false); isFocused = signal(false); isDraggingOver = signal(false); @@ -219,7 +223,7 @@ export class ChatInputComponent { // Clear input and pending uploads this.userInput.set(''); - this.isExpanded.set(false); + this.resetTextareaHeight(); this.fileUploadService.clearReadyUploads(); } @@ -296,10 +300,29 @@ export class ChatInputComponent { onTextareaInput(event: Event) { const textarea = event.target as HTMLTextAreaElement; this.userInput.set(textarea.value); - - // Auto-expand based on content + this.autoResize(textarea); + } + + /** + * Grow the textarea with its content up to MAX_TEXTAREA_HEIGHT_PX, past which + * it scrolls internally (the template sets overflow-y-auto). Without the clamp + * the inline height keeps growing past max-height and the scrollbar never + * becomes usable. + */ + private autoResize(textarea: HTMLTextAreaElement): void { textarea.style.height = 'auto'; - textarea.style.height = textarea.scrollHeight + 'px'; + const height = Math.min(textarea.scrollHeight, MAX_TEXTAREA_HEIGHT_PX); + textarea.style.height = `${height}px`; + } + + /** Collapse the textarea back to a single row (after submit or clear). */ + private resetTextareaHeight(): void { + const textarea = this.messageInput()?.nativeElement; + if (!textarea) { + return; + } + textarea.style.height = `${MIN_TEXTAREA_HEIGHT_PX}px`; + textarea.scrollTop = 0; } onKeyDown(event: KeyboardEvent) { diff --git a/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts b/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts index b17ad3c58..740ff8d23 100644 --- a/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts +++ b/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts @@ -310,6 +310,12 @@ export class InferenceAgentCoreConstruct extends Construct { DYNAMODB_MANAGED_MODELS_TABLE_NAME: managedModelsTableName, DYNAMODB_USER_SETTINGS_TABLE_NAME: userSettingsTableName, DYNAMODB_USER_FILES_TABLE_NAME: userFilesTableName, + // Bucket the runtime writes generated Word docs to (create/modify + // word-document tools). Without this the tool falls back to the + // literal "user-files" default and PutObject is AccessDenied. The + // runtime role's UserFilesBucketAccess statement already grants + // Get/Put/Delete/List on this bucket. + S3_USER_FILES_BUCKET_NAME: props.refs.fileUploadBucket.bucketName, DYNAMODB_SYSTEM_PROMPTS_TABLE_NAME: systemPromptsTableName, // Auth providers diff --git a/infrastructure/lib/constructs/observability/prompt-cache-observability-construct.ts b/infrastructure/lib/constructs/observability/prompt-cache-observability-construct.ts new file mode 100644 index 000000000..4f0706798 --- /dev/null +++ b/infrastructure/lib/constructs/observability/prompt-cache-observability-construct.ts @@ -0,0 +1,198 @@ +import * as cdk from 'aws-cdk-lib'; +import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch'; +import { Construct } from 'constructs'; + +import { AppConfig, getResourceName } from '../../config'; + +export interface PromptCacheObservabilityConstructProps { + config: AppConfig; +} + +/** + * PromptCacheObservabilityConstruct — fleet-wide dashboard + alarms for + * the prompt-cache EMF metrics emitted by + * `backend/src/apis/shared/observability/emf.py` (PR #697). + * + * The metrics arrive as raw EMF JSON on stdout from BOTH compute + * surfaces (inference-api via the AgentCore Runtime log group, app-api + * via ECS awslogs) into the shared `AgentCoreStack/PromptCache` + * namespace — the default of the `EMF_NAMESPACE` env var, which is + * deliberately NOT set in CDK (dev and prod are separate AWS accounts, + * so the unscoped namespace never collides). All metrics are + * dimension-less and designed for `Sum`; per-call detail (`modelId`, + * `sessionId`, `cacheStatus`) rides as queryable EMF log properties, + * hence the Logs Insights widget below rather than dimension fan-out. + * + * This is the first cross-service dashboard, so it lives in its own + * construct area instead of inside the inference-api construct. The + * per-session drill-down counterpart is the cost-anatomy admin endpoint + * (`GET /admin/costs/sessions/{id}/calls`). + * + * Alarms are console-only — no SNS topics exist in the stack yet (same + * posture as kb-sync and scheduled-runs). They use NOT_BREACHING for + * missing data because the `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` + * kill switch (or simply zero traffic) makes the metrics absent + * entirely. + */ +export class PromptCacheObservabilityConstruct extends Construct { + constructor( + scope: Construct, + id: string, + props: PromptCacheObservabilityConstructProps, + ) { + super(scope, id); + + const { config } = props; + + // Must match the default of EMF_NAMESPACE in emf.py. + const namespace = 'AgentCoreStack/PromptCache'; + + const cacheReadTokensMetric = new cloudwatch.Metric({ + namespace, + metricName: 'CacheReadTokens', + statistic: 'Sum', + period: cdk.Duration.minutes(5), + }); + + const cacheWriteTokensMetric = new cloudwatch.Metric({ + namespace, + metricName: 'CacheWriteTokens', + statistic: 'Sum', + period: cdk.Duration.minutes(5), + }); + + const avoidableMissMetric = new cloudwatch.Metric({ + namespace, + metricName: 'AvoidableMiss', + statistic: 'Sum', + period: cdk.Duration.minutes(5), + }); + + const wastedUsdMetric = new cloudwatch.Metric({ + namespace, + metricName: 'WastedUsd', + statistic: 'Sum', + period: cdk.Duration.minutes(5), + }); + + // Cache efficiency: fraction of cache traffic served from cache. + // Healthy steady-state conversations read far more than they write, + // so this should sit near 100%; a prefix-stability regression shows + // up as a sustained drop (writes displacing reads). + const cacheEfficiencyExpression = new cloudwatch.MathExpression({ + expression: '100 * reads / (reads + writes)', + usingMetrics: { + reads: cacheReadTokensMetric, + writes: cacheWriteTokensMetric, + }, + label: 'Cache efficiency (%)', + period: cdk.Duration.minutes(5), + }); + + // ============================================================ + // Dashboard + // ============================================================ + + const dashboard = new cloudwatch.Dashboard(this, 'PromptCacheDashboard', { + dashboardName: getResourceName(config, 'prompt-cache-observability'), + defaultInterval: cdk.Duration.hours(3), + }); + + dashboard.addWidgets( + new cloudwatch.TextWidget({ + markdown: `# Prompt Cache Observability\n**Project:** ${config.projectPrefix} | **Region:** ${config.awsRegion} — fleet-wide EMF metrics from app-api + inference-api. Per-session drill-down: \`GET /admin/costs/sessions/{id}/calls\`.`, + width: 24, + height: 1, + }), + ); + + dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Cache Tokens (Read vs Write)', + left: [cacheReadTokensMetric, cacheWriteTokensMetric], + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'Cache Efficiency (reads / total cache traffic)', + left: [cacheEfficiencyExpression], + leftYAxis: { min: 0, max: 100 }, + width: 12, + height: 6, + }), + ); + + dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Avoidable Cache Misses', + left: [avoidableMissMetric], + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'Wasted USD (avoidable re-writes)', + left: [wastedUsdMetric], + width: 12, + height: 6, + }), + ); + + dashboard.addWidgets( + new cloudwatch.LogQueryWidget({ + title: 'Model Calls by cacheStatus (inference-api)', + // The runtime log group is defined by the inference-api + // construct with this deterministic name; referenced by name + // (not typed ref) since a dashboard widget creates no CFN + // dependency. app-api's ECS log group is auto-named, so its + // (much smaller) share of emissions isn't queried here. + logGroupNames: [`/aws/bedrock-agentcore/runtimes/${config.projectPrefix}`], + queryLines: [ + 'filter ispresent(cacheStatus)', + 'stats count(*) as calls by cacheStatus', + 'sort calls desc', + ], + width: 24, + height: 6, + }), + ); + + // ============================================================ + // Alarms (console-only; no SNS wiring yet) + // ============================================================ + + // AvoidableMiss is the nominated alarm target (see emf.py): a + // prefix-stability regression flips a large share of calls to + // `miss_avoidable`, showing up as a step change in this sum. + new cloudwatch.Alarm(this, 'PromptCacheAvoidableMissAlarm', { + alarmName: getResourceName(config, 'prompt-cache-avoidable-miss'), + alarmDescription: + 'Avoidable prompt-cache misses exceeded threshold — likely a prompt-prefix stability regression', + metric: avoidableMissMetric, + threshold: config.production ? 10 : 50, + evaluationPeriods: 3, + comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD, + treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING, + }); + + new cloudwatch.Alarm(this, 'PromptCacheWastedUsdAlarm', { + alarmName: getResourceName(config, 'prompt-cache-wasted-usd'), + alarmDescription: + 'Dollars wasted on avoidable prompt-cache re-writes exceeded threshold', + metric: wastedUsdMetric, + threshold: config.production ? 1 : 5, + evaluationPeriods: 3, + comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD, + treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING, + }); + + // ============================================================ + // Outputs + // ============================================================ + + new cdk.CfnOutput(this, 'PromptCacheDashboardName', { + value: dashboard.dashboardName, + description: 'CloudWatch Dashboard for prompt-cache observability', + exportName: `${config.projectPrefix}-PromptCacheDashboard`, + }); + } +} diff --git a/infrastructure/lib/platform-stack.ts b/infrastructure/lib/platform-stack.ts index 48bfecd49..5ed350997 100644 --- a/infrastructure/lib/platform-stack.ts +++ b/infrastructure/lib/platform-stack.ts @@ -70,6 +70,9 @@ import { AgentCoreGatewayConstruct } from './constructs/gateway/agentcore-gatewa import { McpSandboxBucketConstruct } from './constructs/mcp-sandbox/mcp-sandbox-bucket-construct'; import { McpSandboxDistributionConstruct } from './constructs/mcp-sandbox/mcp-sandbox-distribution-construct'; +// Observability (cross-service dashboards + alarms) +import { PromptCacheObservabilityConstruct } from './constructs/observability/prompt-cache-observability-construct'; + // Fine-tuning (data half lives in Platform) import { FineTuningDataConstruct } from './constructs/fine-tuning/fine-tuning-data-construct'; @@ -630,6 +633,15 @@ export class PlatformStack extends cdk.Stack { frontendUrl, documentsBucket: this.ragDocumentsBucket, }); + + // ============================================================ + // Prompt-cache observability — cross-service dashboard + alarms + // over the EMF metrics both APIs emit (PR #697 follow-up). + // Metric-namespace-based, so no typed refs are needed. + // ============================================================ + new PromptCacheObservabilityConstruct(this, 'PromptCacheObservability', { + config, + }); } /** diff --git a/infrastructure/package-lock.json b/infrastructure/package-lock.json index a5ac74c81..d9666af9b 100644 --- a/infrastructure/package-lock.json +++ b/infrastructure/package-lock.json @@ -1,12 +1,12 @@ { "name": "infrastructure", - "version": "1.8.0", + "version": "1.9.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "infrastructure", - "version": "1.8.0", + "version": "1.9.0", "dependencies": { "aws-cdk-lib": "2.260.0", "constructs": "10.6.0" diff --git a/infrastructure/package.json b/infrastructure/package.json index 75781bdcc..d12452285 100644 --- a/infrastructure/package.json +++ b/infrastructure/package.json @@ -1,6 +1,6 @@ { "name": "infrastructure", - "version": "1.8.0", + "version": "1.9.0", "bin": { "infrastructure": "bin/infrastructure.js" }, diff --git a/infrastructure/test/prompt-cache-observability.test.ts b/infrastructure/test/prompt-cache-observability.test.ts new file mode 100644 index 000000000..75aebfc74 --- /dev/null +++ b/infrastructure/test/prompt-cache-observability.test.ts @@ -0,0 +1,86 @@ +import * as cdk from 'aws-cdk-lib'; +import { Template } from 'aws-cdk-lib/assertions'; + +import { PromptCacheObservabilityConstruct } from '../lib/constructs/observability/prompt-cache-observability-construct'; +import { createMockConfig, MOCK_ACCOUNT, MOCK_PREFIX, MOCK_REGION } from './helpers/mock-config'; + +function synth(production: boolean): Template { + const app = new cdk.App(); + const stack = new cdk.Stack(app, 'Test', { + env: { account: MOCK_ACCOUNT, region: MOCK_REGION }, + }); + const config = createMockConfig({ production }); + new PromptCacheObservabilityConstruct(stack, 'PromptCacheObservability', { + config, + }); + return Template.fromStack(stack); +} + +describe('PromptCacheObservabilityConstruct', () => { + let t: Template; + beforeAll(() => { + t = synth(false); + }); + + it('creates the dashboard with the conventional name', () => { + t.hasResourceProperties('AWS::CloudWatch::Dashboard', { + DashboardName: `${MOCK_PREFIX}-prompt-cache-observability`, + }); + }); + + it('dashboard graphs the EMF namespace metrics and queries the runtime log group', () => { + const dashboards = t.findResources('AWS::CloudWatch::Dashboard'); + const body = JSON.stringify(Object.values(dashboards)[0].Properties.DashboardBody); + for (const metric of ['CacheReadTokens', 'CacheWriteTokens', 'AvoidableMiss', 'WastedUsd']) { + expect(body).toContain(metric); + } + expect(body).toContain('AgentCoreStack/PromptCache'); + expect(body).toContain(`/aws/bedrock-agentcore/runtimes/${MOCK_PREFIX}`); + expect(body).toContain('cacheStatus'); + }); + + it('creates exactly two alarms, both NOT_BREACHING on missing data (kill-switch tolerant)', () => { + t.resourceCountIs('AWS::CloudWatch::Alarm', 2); + const alarms = Object.values(t.findResources('AWS::CloudWatch::Alarm')); + for (const alarm of alarms) { + expect(alarm.Properties.TreatMissingData).toBe('notBreaching'); + expect(alarm.Properties.Namespace).toBe('AgentCoreStack/PromptCache'); + expect(alarm.Properties.Statistic).toBe('Sum'); + // Console-only: no SNS wiring yet anywhere in the stack. + expect(alarm.Properties.AlarmActions).toBeUndefined(); + } + }); + + it('alarms on AvoidableMiss (the nominated target) and WastedUsd', () => { + t.hasResourceProperties('AWS::CloudWatch::Alarm', { + AlarmName: `${MOCK_PREFIX}-prompt-cache-avoidable-miss`, + MetricName: 'AvoidableMiss', + Threshold: 50, + EvaluationPeriods: 3, + ComparisonOperator: 'GreaterThanThreshold', + }); + t.hasResourceProperties('AWS::CloudWatch::Alarm', { + AlarmName: `${MOCK_PREFIX}-prompt-cache-wasted-usd`, + MetricName: 'WastedUsd', + Threshold: 5, + }); + }); + + it('uses stricter thresholds in production', () => { + const prod = synth(true); + prod.hasResourceProperties('AWS::CloudWatch::Alarm', { + MetricName: 'AvoidableMiss', + Threshold: 10, + }); + prod.hasResourceProperties('AWS::CloudWatch::Alarm', { + MetricName: 'WastedUsd', + Threshold: 1, + }); + }); + + it('exports the dashboard name', () => { + t.hasOutput('*', { + Export: { Name: `${MOCK_PREFIX}-PromptCacheDashboard` }, + }); + }); +});