fix(reporting): resolve SkillReport.model from the actual API response - #472
Open
CalebKAston wants to merge 1 commit into
Open
fix(reporting): resolve SkillReport.model from the actual API response#472CalebKAston wants to merge 1 commit into
CalebKAston wants to merge 1 commit into
Conversation
SkillReport.model only ever echoed the caller-configured model (options.model / trigger.model). When no model override is configured at any of the 4 levels a run can set one (per-skill, defaults.agent, defaults, per-trigger), this field is always undefined — even though the runtime already knows exactly which model answered each hunk. The runtime normalizes this today as SkillRunResult.responseModel (sourced from the live API response, see runtimes/claude.ts's normalizeResult), but it only ever fed an opt-in HunkTrace object (captureTraces) and OTel spans — never the exported SkillReport.model consumers actually read from the findings-file. This threads responseModel from analyzeHunk's return paths that have a live API response (including the circuit-breaker-opens-on-an-SDK-error path via hunkFailureFromCircuit, which previously had the value available but dropped it) up through analyzeFile and both independent report-construction sites: sdk/analyze.ts's runSkillAnalysis and the CLI's separate tasks.ts pipeline (tasks.ts drives analyzeFile directly with its own aggregation rather than calling runSkillAnalysis, so both needed the same treatment). Collapses to a single value via a new resolveResponseModel(models, fallback) helper in usage.ts when every hunk in the run agrees, falling back to the configured override when hunks disagree or none reported one — mirroring the existing model/models singular-vs-plural collapse pattern already used for auxiliaryUsageAttribution. No schema change: SkillReportSchema.model already accepted a string, this only changes which value populates it. Adds regression coverage for: the single-model case, the fallback-on-disagreement case, the circuit-breaker-with-a-live-response case, and — since this value is computed per independent skill execution — that concurrent skills each get their own model correctly attributed rather than any cross-skill bleed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SkillReport.modelonly ever echoes the caller-configured model (options.model/trigger.model). When no model override is configured at any of the 4 levels a run can set one (per-skill,defaults.agent,defaults, per-trigger), this field is alwaysundefined— even though the runtime already knows exactly which model answered each hunk.The runtime normalizes this today as
SkillRunResult.responseModel(sourced from the live API response, seeruntimes/claude.ts'snormalizeResult), but it only ever fed an opt-inHunkTraceobject (captureTraces) and OTel spans — never the exportedSkillReport.modelthat findings-file consumers actually read.This threads
responseModelfrom everyanalyzeHunkreturn path that has a live API response — including the circuit-breaker-opens-on-an-SDK-error path viahunkFailureFromCircuit, which now accepts and forwards it — up throughanalyzeFileand both independent report-construction sites:sdk/analyze.ts'srunSkillAnalysisand the CLI's separatetasks.tspipeline (tasks.tsdrivesanalyzeFiledirectly with its own aggregation rather than callingrunSkillAnalysis, so both need the same treatment).Collapses to a single value via a new
resolveResponseModel(models, fallback)helper inusage.tswhen every hunk in the run agrees, falling back to the configured override when hunks disagree or none reported one — mirroring the existing model/models singular-vs-plural collapse pattern already used forauxiliaryUsageAttribution.No schema change:
SkillReportSchema.modelalready accepted a string, this only changes which value populates it.Test plan
pnpm lint && pnpm build && pnpm test— all green (93 test files, 1832 passing, 4 pre-existing skips)