None of the 2,667 FailureSensorIQ questions reference an entity in the loaded database

[arthapraha]

Repository: IBM/AssetOpsBench · Observed at: e11d1c1 (current main)
Dataset: ibm-research/AssetOpsBench at revision 5e25bb7f2cd37fb68b9a9e1f99d170ca5be7ce17
Source: finding 6 of our defect report · follows #503, completes the set with #506, #507 and #508

The MCQA pools carry answer keys, which makes them by far the most attractive
material in the benchmark for automated evaluation. They are also the one part
that never touches the environment the benchmark loads.

This is not a defect and we are not asking for a change to the questions. It
is a caveat we would have valued knowing before we started.

The measurement

The claim holds under every defensible definition of the denominator — which
matters, because "how many MCQA questions are there" turns out to be a judgement
about which files to count, not a lookup.

file or union	questions	exact rule	normalised rule
failuresensoriq_standard/all.jsonl	2,667	0	0
failuresensoriq_standard/all_10_options.jsonl	2,667	0	0
failuresensoriq_standard/all_multi_answers.jsonl	5,629	0	0
failuresensoriq_standard/sample_50_questions.jsonl	50	0	0
failuresensoriq_perturbed/…_all_perturbed_simple.jsonl	2,667	0	0
failuresensoriq_perturbed/…_perturbed_complex.jsonl	2,667	0	0
standard set, deduplicated by question text	2,256	0	0
standard + perturbed, deduplicated	4,141	0	0

Entities searched — everything the shipped collections actually contain, from
src/couchdb/scenarios_data/shared/:

Chiller 6, CHILLER6, Motor_01, PUMP3, AHU2, mp_1, hyd_1, MAIN,
NORTH, RIVPLANT, 1000045, 1000046, 1000050, ttm_96_28

Searched across each question's stem, all of its options, and its asset_name
field. Two rules, because one was not enough:

• exact — literal substring.
• normalised — separators inside an entity are optional, so Chiller 6
  matches CHILLER6, but the match must begin and end at a word boundary.

A false positive we hit ourselves, stated because you would hit it too

Our first normalised rule dropped word boundaries entirely. It reported six
hits — every one of them the site id MAIN matching inside the word
"maintenance":

"In the context of steam turbine maintenance, which of the available sensors…"

We nearly published that as a finding. It is exactly the trap this kind of
counting sets: MAIN and NORTH are site identifiers that are also ordinary
English, so a separator-insensitive substring rule manufactures hits out of
prose. The fix is the word boundary, which keeps the one case a loose rule exists
for — Chiller 6 ≡ CHILLER6, the spelling split reported in #507 — and drops
the rest.

We mention it because anyone re-running this with a looser rule will get six
hits and reasonably think we were wrong. Those six are the reason to require a
boundary, not evidence of environment coupling.

Why this is worth a line in the docs

The questions are about equipment classes and failure physics — "for an electric
motor, if a rotor windings fault occurs, which sensor is most relevant". They
are good questions of that kind. But:

• A system scored on this pool is not being scored on tool use. No retrieval
  from the loaded database can help, because nothing in the pool refers to it.
• An experiment that varies tool access will find no effect here, and the
  null result will look like a finding about the agent rather than about the
  question set. That is the specific way this costs someone a day.
• Combined with 14 of 55 scenario questions can be answered from the shipped data — the audit, with dispositions #507 — 14 of 55 scenario questions gradeable against the shipped
  data — the natively-scoreable pool and the environment-exercising pool are
  nearly disjoint. That is the sentence we would have wanted up front.

What we are asking for

Nothing in the code, and no change to the questions. A line in
docs/evaluation.md — or wherever you prefer — saying what the MCQA pools do and
do not exercise, so a user choosing them for automated evaluation knows they are
measuring domain knowledge rather than asset operations.

The full script is available, and we are happy to contribute it, rerun it
against a different entity list, or be shown that our entity list is the wrong
one — that last outcome would be the most useful of the three.

Relationship to the other issues

This is the last of four issues closed by a single documentation PR, after
#506, #507 and #508. The PR follows immediately now that all four have
numbers, as promised in #506.

The content is written; its location is your call.

---

Findings 1–6 reached the maintainers by email on 10 August. This continues that
exchange, and is the fifth of the six issues announced in #503 — the sixth
follows, with a code PR.