`static_json` scores differently-named assets as perfect matches when their names share a digit

[arthapraha]

Repository: IBM/AssetOpsBench · Observed at: e11d1c1 (current main)
Source: finding 2 of our defect report · follows #503

What happens

parse_structured_answer cannot parse "Chiller 6" as JSON, so it falls
through to _extract_final_count_from_text, which delegates first to
_extract_count_from_text — and that accepts any answer containing exactly one
number, wherever the number sits. "Chiller 6" therefore parses as 6.

So does "Boiler 6". The two then compare equal:

gold ["Chiller 6"]  vs  model ["Boiler 6"]   → strict 1.000  PASSED
gold ["Chiller 6"]  vs  model ["Chiller 4"]  → strict 0.000  failed

A wrong asset passes; a different wrong asset fails. The grader is keying on
a digit that happens to be inside a name.

This inflates scores rather than deflating them, which is why we would fix it
early: a failing grader is noticed, a generous one is not.

Observed live, not constructed

Running scenario 3 through the repository's own plan_execute runner, the
pipeline reduced a correct answer —

"The MAIN site has five assets: Chiller 6, Motor_01, PUMP3, hyd_1, and mp_1."

— to {'answer': '6'}. It failed against the correct gold, and would have
passed against a gold of 6, which is the answer to nothing.

Reproduce

from evaluation.scorers.static_json import (
    parse_structured_answer as p,
    _extract_count_from_text as f,
)

f("Chiller 6")   # -> 6   root cause: a digit inside a name reads as a count
f("Boiler 6")    # -> 6   so the two answers are indistinguishable
p("Chiller 6")   # -> 6   and that is the value the scorer actually compares

Related open issues — same file, opposite direction

#494 (open, self-assigned) reports the static JSON evaluator extracting an
earlier parenthesised phrase instead of the final numeric answer, so correct
answers fail. #475 is the MCQ variant.

Those are false negatives; this is a false positive. Same extraction path,
opposite direction: #494 loses correct answers, this one accepts wrong ones. A
fix for either that treats the extractor as "harvest a number from the text"
will keep the other. We would rather this be folded into whatever #494's
assignee is already doing than land beside it — happy to rework the patch to
fit that shape, or to close this in favour of a combined change.

Suggested fix

The docstring already states the intent — "count-only or nearly count-only" —
and the fix enforces it rather than changing it. Once the number and the usual
count wording are removed, any remaining word means the number was part of a
phrase, most importantly part of a name.

The change lands in _extract_count_from_text, so it covers both entry points —
the direct call and the delegation from _extract_final_count_from_text.
"The count is 6", "There are 5 assets" and "Answer: 6" still parse as
counts. "Chiller 6" no longer does: parse_structured_answer returns the
string, and the answer is compared as a name rather than as a number.

Patch is ready — 2 files, +50/-1, with tests — and we will open the PR against
this issue as soon as it has a number.

Note for reviewers, unrelated to this change

6 tests in src/evaluation/tests/ already fail on main at e11d1c1 —
test_car_metadata_* and test_static_json_scorer_uses_car_metadata_score.
This change does not touch them. The suite goes from 6 failed / 90 passed to
6 failed / 92 passed. Flagging it so a red CI run on this PR is not read as
this PR's doing; we are happy to file it separately if useful.