feat(metrics): Adherence To Prompt

[Ayush8923]

Issue

Closes #1046

Summary

Adds the second judge metric, Adherence to Prompt — to the v2 native LLM-as-Judge, and simplifies how judge results are stored.
Before: the v2 judge scored one metric (Adherence to Ground Truth), and each metric wrote its own per-row DB column.
Now: one combined call per row returns both metrics, each explicitly scoped to its own inputs, and per-row results live in a single place (the run's trace file) instead of duplicate DB columns.

1. New metric: Adherence to Prompt

• Scores whether the answer obeys the evaluated bot own configured instructions, not factual correctness. One holistic 0–1 score + reasoning across four dimensions: language/tone, answer-vs-refuse, fallback-vs-fabrication, and injection resistance.

2. Still one combined call, now explicitly scoped

• Both metrics are graded by a single call per row returning one JSON keyed by metric. Because every input block is visible to every metric in a combined call, each metric section now carries registry-derived scoping lines:
  • When scoring THIS metric, consider only these input blocks: .
  • Do not consider: .

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Summary by CodeRabbit

• New Features

  • Added native LLM-as-a-judge scoring for adherence to configured prompts, including the “Adherence to Prompt” metric.
  • Judge runs now support combined judging with shared rubric inputs, including configured instructions and prompt templates when available.
  • Evaluation runs now expose an is_judge_run flag to clarify judge-run behavior.

• Bug Fixes

  • Improved robustness so malformed judge outputs only drop the affected row’s metrics, without breaking other results.

• Documentation

  • Updated API and cost documentation to reflect the new judged-run and cost accounting behavior.

• Chores

  • Updated/expanded automated tests for judged-run scoring and gating.

[github-actions]

OpenAPI changes   🟢 5 non-breaking changes

Tip

Safe to merge from an API-contract perspective.

Full changelog  ·  5

	Method	Path	Change
🟢	GET	/api/v1/evaluations	added the optional property data/anyOf[subschema #1]/items/is_judge_run to the response with the 200 status
🟢	POST	/api/v1/evaluations	added the optional property data/anyOf[subschema #1: EvaluationRunPublic]/is_judge_run to the response with the 200 status
🟢	GET	/api/v1/evaluations/{evaluation_id}	added the optional property data/anyOf[subschema #1: EvaluationRunPublic]/is_judge_run to the response with the 200 status
🟢	POST	/api/v2/evaluations	endpoint added
🟢	POST	/api/v2/evaluations/datasets	endpoint added

_{main ↔ 3510e3bc · generated by oasdiff}

[codecov]

Codecov Report

❌ Patch coverage is 97.72727% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
backend/app/crud/evaluations/fast.py	86.20%	4 Missing ⚠️
...kend/app/tests/crud/evaluations/test_fast_judge.py	99.26%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Ayush8923]

Now, this column is not needed. because this column is not used anywhere.

[coderabbitai]

📝 Walkthrough

Walkthrough

V2 evaluation runs now perform combined native judge scoring with a new prompt-adherence metric. Configuration prompts gate metric availability, judge costs use one judge stage, and durable per_item_ground_truth storage is replaced by is_judge_run.

Changes

Native judge evaluation flow

Layer / File(s)	Summary
Judge run persistence contracts backend/app/alembic/versions/075_...py, backend/app/models/evaluation.py, backend/app/api/routes/evaluations/*_v2.py	The schema and API models remove per_item_ground_truth, retain is_judge_run, and document current v2 behavior.
Combined judge metric composition backend/app/crud/evaluations/judge.py, backend/app/crud/evaluations/score.py	A prompt-adherence metric is added, metric selection is gated by available inputs, and combined judge prompts include scoped rubrics and configuration prompts.
Fast judged-run orchestration backend/app/crud/evaluations/fast.py, backend/app/crud/evaluations/cost.py	Stage 3 resolves configuration prompts, executes combined row judging, records one judge cost stage, emits trace summaries, and persists cost and unscoreable state.
Judge and pipeline validation backend/app/tests/crud/evaluations/*, backend/app/tests/api/routes/test_evaluation_v2.py	Tests cover prompt scoring, input composition, metric gating, malformed-row isolation, cost attribution, and unchanged v1 behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant EvaluationRun
  participant Stage3
  participant ConfigResolver
  participant JudgeRow
  Stage3->>ConfigResolver: resolve instructions and prompt template
  ConfigResolver-->>Stage3: config_prompt or unavailable
  Stage3->>JudgeRow: judge row with selected metrics
  JudgeRow-->>Stage3: combined ground-truth and prompt scores
  Stage3->>EvaluationRun: persist cost and unscoreable state

Loading

Possibly related PRs

• ProjectTech4DevAI/kaapi-backend#1004: Continues the native LLM judge flow refactored here.
• ProjectTech4DevAI/kaapi-backend#1055: Overlaps on native judge execution, schema, and judge-run persistence.
• ProjectTech4DevAI/kaapi-backend#870: Introduces the fast evaluation pipeline modified here.

Suggested labels: enhancement

Suggested reviewers: akhileshnegi, vprashrex

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.09% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR implements the requested prompt-adherence judge metric and covers the listed behaviors in scoring and tests.
Out of Scope Changes check	✅ Passed	The changes stay focused on prompt-adherence judging, supporting model, cost, and test updates without clear unrelated scope.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding the Adherence to Prompt metric.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/adherence-to-prompt

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

backend/app/crud/evaluations/judge.py (1)

285-317: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject judge responses that omit any enabled metric.

Line 312 accepts partial JSON, so a response containing only ground_truth silently drops prompt without marking the row failed. This produces biased trace and summary coverage. Validate every enabled key and raise ValueError when one is missing.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/crud/evaluations/judge.py` around lines 285 - 317, Update
_parse_judge_output to validate that every enabled metric in metrics has a
corresponding key in the parsed data object before returning results. Raise
ValueError when any metric is missing, while preserving the existing score
parsing for complete responses and invalid-value handling.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/app/crud/evaluations/cost.py`:
- Around line 118-119: Update the eval_run.cost carry-forward logic for writing
the combined judge stage so legacy per-metric keys such as ground_truth_judge
are removed before judge is persisted. Ensure deployment-retry and
partial-update flows retain only the combined judge entry and add coverage for
this behavior.

In `@backend/app/crud/evaluations/fast.py`:
- Around line 783-793: Update the prompt-section construction around the prompt
template check so the template is stripped before being appended. Only add the
PROMPT_TEMPLATE_LABEL section when the stripped template content is non-empty,
ensuring whitespace-only templates do not produce scoring input; preserve the
existing instructions handling and section joining.
- Around line 1064-1068: Update the judgeable-row filtering around the
response-results comprehension so prompt-adherence rows are retained when
generated_output exists even if ground_truth is empty. Separate row-level metric
selection from ground-truth-dependent metrics, and invoke the combined scoring
path with only the prompt-adherence metric when ground truth is unavailable,
while preserving existing behavior for metrics that require ground truth.

In `@backend/app/crud/evaluations/judge.py`:
- Around line 113-123: Rename enabled_metric_specs to list_enabled_metric_specs
to follow the repository’s plural-fetch naming convention, preserving its
filtering and return behavior. Update every caller, including orchestration code
and tests, to use the renamed function.

In `@backend/app/tests/crud/evaluations/test_fast_judge.py`:
- Around line 158-163: Complete the callable annotations across the affected
sites: in backend/app/tests/crud/evaluations/test_fast_judge.py lines 158-163,
annotate usage and add a precise return type to _both_metrics_response; in lines
326-800, add -> None to changed test methods and type all local judge callback
parameters and returns; in
backend/app/alembic/versions/075_add_judge_columns_to_evaluation_run.py lines
20-36, add -> None to upgrade and downgrade; and in
backend/app/tests/crud/evaluations/test_judge.py lines 56-60, annotate the
helper return as list[JudgeMetricSpec]. Use narrowed concrete types rather than
Any.

---

Outside diff comments:
In `@backend/app/crud/evaluations/judge.py`:
- Around line 285-317: Update _parse_judge_output to validate that every enabled
metric in metrics has a corresponding key in the parsed data object before
returning results. Raise ValueError when any metric is missing, while preserving
the existing score parsing for complete responses and invalid-value handling.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1f11cae0-4935-45bb-b4bd-f0c2e878cd76

📥 Commits

Reviewing files that changed from the base of the PR and between b6328ed and 6e3c994.

📒 Files selected for processing (11)

• backend/app/alembic/versions/075_add_judge_columns_to_evaluation_run.py
• backend/app/api/routes/evaluations/dataset_v2.py
• backend/app/api/routes/evaluations/evaluation_v2.py
• backend/app/crud/evaluations/cost.py
• backend/app/crud/evaluations/fast.py
• backend/app/crud/evaluations/judge.py
• backend/app/crud/evaluations/score.py
• backend/app/models/evaluation.py
• backend/app/tests/api/routes/test_evaluation_v2.py
• backend/app/tests/crud/evaluations/test_fast_judge.py
• backend/app/tests/crud/evaluations/test_judge.py

💤 Files with no reviewable changes (1)

• backend/app/models/evaluation.py