Typed criterion verdicts on grading results

[solvemproblr]

Issue

Rubric-style graders had no structured way to report per-criterion verdicts. LLMJudgeGrader buried them in SubScore.metadata as an ad-hoc dict keyed by truncated criterion text, and since metadata is excluded from serialization, the only way verdicts reached the platform was via combine() copying metadata into the free-form EvaluationResult.info. Environments worked around this by stuffing custom rubric dicts into info, so the trace viewer could only show raw JSON.

Solution

• New CriterionResult model (criterion, passed, weight, reason, source) exported from hud.graders.
• SubScore gains a serialized criteria field — verdicts ride on the subscore of the grader that checked them, and combine() carries them through into EvaluationResult.subscores.
• LLMJudgeGrader emits CriterionResults directly (replacing the old truncated-key report in metadata). Custom graders return them under the "criteria" metadata key, which Grader.grade() promotes to the typed field.
• combine_any/combine_all merge input criteria onto the collapsed subscore, tagging source with the input's name so attribution survives the collapse.

Outcome

Criterion verdicts are now typed end to end and land on the wire at evaluation_result.subscores[].criteria, so the trace viewer can render rubrics structurally (per-subscore verdict lists with reasons) instead of an info JSON dump. Breaking: the judge's criteria report no longer appears in info.

---

Note

Medium Risk
Changes the public grading/trace payload shape with a documented breaking move away from info for judge criteria; scope is evaluation/serialization with solid test coverage, not security-critical paths.

Overview
Introduces CriterionResult and wires per-criterion verdicts through grading so they serialize on evaluation_result.subscores[].criteria instead of ad-hoc metadata or info JSON.

SubScore now has a serialized criteria list. Grader.grade() lifts a "criteria" key from compute metadata into that field; LLMJudgeGrader emits CriterionResult objects directly (replacing the old truncated-key report in metadata). combine keeps criteria on each subscore; combine_any / combine_all merge input criteria onto the collapsed subscore and set source for attribution.

Docs export CriterionResult and describe the rubric pattern. Breaking: LLM judge rubric details no longer land in EvaluationResult.info—use subscores[].criteria. Unrelated fix: OpenAI tool empty output uses ResponseInputTextContentParam.

^{Reviewed by Cursor Bugbot for commit bb711af. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
hud	🟢 Ready	View Preview	Jul 9, 2026, 11:40 PM