Skip to content

Make golden-diff evaluations reproducible across providers #74

Description

@Scriptception

Problem

Golden fixtures currently invoke a real agent manually, so results depend on local configuration and cannot be reproduced consistently across model providers or harnesses.

User outcome

Maintainers can run the same declared evaluation protocol and compare complete results across supported backends.

Scope

  • Define a provider-neutral evaluation-run schema and adapter interface.
  • Record model identity, harness version, configuration, seed where available, run count, failures, duration, and token usage where exposed.
  • Support dry-run fixture validation and explicit cost limits.
  • Store raw evaluation artifacts outside the public summary when they contain sensitive content.

Non-goals

  • Hiding provider differences behind one unsupported aggregate.
  • Requiring paid evaluation for every contributor pull request.

Acceptance criteria

  • At least two harness adapters can execute the same fixture contract.
  • Every attempted run and incomplete result is recorded.
  • Budgets and concurrency are bounded before execution.
  • Results can be reproduced from an immutable skill, fixture, and run configuration.

Validation

  • Run repeated evaluations on a representative subset and report variance.
  • Replay a stored configuration and verify schema-equivalent output.

Relationships

  • Precedes Track evaluation baselines and detect quality regressions.
  • Uses author validation and fixture integrity checks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions