Skip to content

Evals: per-PR gate is structurally near-circular and pass_rate_strict has no regression gate #68

Description

@JumpTechCode

Summary

The per-PR eval gate replays committed cassettes against a baseline that was itself generated by replaying those same cassettes, so per-case diffs are ~0 by construction — the paired bootstrap mainly fires on scoring/runner code changes, not model-quality drift (drift detection is consciously offloaded to the nightly live-API job). Compounding it, the README headline number pass_rate_strict has no PR-level regression gate at all.

Evidence

  • Makefileevals-baseline (:96) builds evals/baselines/main.json by replaying sentinel/evals/cassettes; evals (:88) + evals-compare (:99, compare-to-baseline) replay the same cassettes against that baseline.
  • .github/workflows/ci.yml:112-199 (evals-gate, SENTINEL_EVAL_CASSETTE_MODE: replay) vs nightly-evals.yml (--live, the actual drift sensor).
  • sentinel/evals/cli.py:973 — the regression verdict iterates only category_match, hypothesis_cosine, action_coverage, evidence_quality. pass_rate_strict lives under headline (report.py:78) and is never read by _compute_regression_verdict (cli.py:951-1050).

Notes

Cassette-replay determinism is a deliberate, documented design choice — this is not "replay is wrong." It's: (a) make the README's "blocks regressions in CI" honest about what the PR gate vs nightly job each catch, and (b) give pass_rate_strict an explicit regression gate.

Fix

  • Add a pass_rate_strict regression gate (nightly at minimum).
  • Tighten README wording to distinguish per-PR (scoring-code / structural) from nightly (model drift).

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingTest coverage gaps or improvements

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions