Skip to content

fix(reporting): resolve SkillReport.model from the actual API response - #472

Open
CalebKAston wants to merge 1 commit into
getsentry:mainfrom
babylist:fix/upstream-observed-response-model
Open

fix(reporting): resolve SkillReport.model from the actual API response#472
CalebKAston wants to merge 1 commit into
getsentry:mainfrom
babylist:fix/upstream-observed-response-model

Conversation

@CalebKAston

@CalebKAston CalebKAston commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

SkillReport.model only ever echoes the caller-configured model (options.model / trigger.model). When no model override is configured at any of the 4 levels a run can set one (per-skill, defaults.agent, defaults, per-trigger), this field is always undefined — even though the runtime already knows exactly which model answered each hunk.

The runtime normalizes this today as SkillRunResult.responseModel (sourced from the live API response, see runtimes/claude.ts's normalizeResult), but it only ever fed an opt-in HunkTrace object (captureTraces) and OTel spans — never the exported SkillReport.model that findings-file consumers actually read.

This threads responseModel from every analyzeHunk return path that has a live API response — including the circuit-breaker-opens-on-an-SDK-error path via hunkFailureFromCircuit, which now accepts and forwards it — up through analyzeFile and both independent report-construction sites: sdk/analyze.ts's runSkillAnalysis and the CLI's separate tasks.ts pipeline (tasks.ts drives analyzeFile directly with its own aggregation rather than calling runSkillAnalysis, so both need the same treatment).

Collapses to a single value via a new resolveResponseModel(models, fallback) helper in usage.ts when every hunk in the run agrees, falling back to the configured override when hunks disagree or none reported one — mirroring the existing model/models singular-vs-plural collapse pattern already used for auxiliaryUsageAttribution.

No schema change: SkillReportSchema.model already accepted a string, this only changes which value populates it.

Test plan

  • pnpm lint && pnpm build && pnpm test — all green (93 test files, 1832 passing, 4 pre-existing skips)
  • Regression coverage for: the single-model case, the fallback-on-disagreement case, the circuit-breaker-with-a-live-response case, and — since this value is computed per independent skill execution — that concurrent skills each get their own model correctly attributed rather than any cross-skill bleed

SkillReport.model only ever echoed the caller-configured model
(options.model / trigger.model). When no model override is configured
at any of the 4 levels a run can set one (per-skill, defaults.agent,
defaults, per-trigger), this field is always undefined — even though
the runtime already knows exactly which model answered each hunk.

The runtime normalizes this today as SkillRunResult.responseModel
(sourced from the live API response, see runtimes/claude.ts's
normalizeResult), but it only ever fed an opt-in HunkTrace object
(captureTraces) and OTel spans — never the exported SkillReport.model
consumers actually read from the findings-file.

This threads responseModel from analyzeHunk's return paths that have
a live API response (including the circuit-breaker-opens-on-an-SDK-error
path via hunkFailureFromCircuit, which previously had the value
available but dropped it) up through analyzeFile and both independent
report-construction sites: sdk/analyze.ts's runSkillAnalysis and the
CLI's separate tasks.ts pipeline (tasks.ts drives analyzeFile directly
with its own aggregation rather than calling runSkillAnalysis, so both
needed the same treatment).

Collapses to a single value via a new resolveResponseModel(models,
fallback) helper in usage.ts when every hunk in the run agrees,
falling back to the configured override when hunks disagree or none
reported one — mirroring the existing model/models singular-vs-plural
collapse pattern already used for auxiliaryUsageAttribution.

No schema change: SkillReportSchema.model already accepted a string,
this only changes which value populates it.

Adds regression coverage for: the single-model case, the
fallback-on-disagreement case, the circuit-breaker-with-a-live-response
case, and — since this value is computed per independent skill
execution — that concurrent skills each get their own model correctly
attributed rather than any cross-skill bleed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant