Problem
Golden fixtures currently invoke a real agent manually, so results depend on local configuration and cannot be reproduced consistently across model providers or harnesses.
User outcome
Maintainers can run the same declared evaluation protocol and compare complete results across supported backends.
Scope
- Define a provider-neutral evaluation-run schema and adapter interface.
- Record model identity, harness version, configuration, seed where available, run count, failures, duration, and token usage where exposed.
- Support dry-run fixture validation and explicit cost limits.
- Store raw evaluation artifacts outside the public summary when they contain sensitive content.
Non-goals
- Hiding provider differences behind one unsupported aggregate.
- Requiring paid evaluation for every contributor pull request.
Acceptance criteria
Validation
- Run repeated evaluations on a representative subset and report variance.
- Replay a stored configuration and verify schema-equivalent output.
Relationships
- Precedes Track evaluation baselines and detect quality regressions.
- Uses author validation and fixture integrity checks.
Problem
Golden fixtures currently invoke a real agent manually, so results depend on local configuration and cannot be reproduced consistently across model providers or harnesses.
User outcome
Maintainers can run the same declared evaluation protocol and compare complete results across supported backends.
Scope
Non-goals
Acceptance criteria
Validation
Relationships