14 of 55 scenario questions can be answered from the shipped data — the audit, with dispositions

[arthapraha]

Repository: IBM/AssetOpsBench · Observed at: e11d1c1 (current main)
Source: findings 8 and 5 of our defect report · follows #503, sibling of #506

#506 asks whether the public artefacts are meant to be scoreable end-to-end.
This issue is the measurement behind that question — we ran the audit before
asking, so the question arrives with a number rather than an impression.

Like #506, this is offered as evidence for a documentation decision, not as a
list of defects. If the answer to #506 is "the public data is a sample, not a
gradeable set", then everything below is expected and the useful outcome is a
sentence in the docs saying so.

What we did

We took every scenario question in the utterance set and asked one thing: can
the correct answer be derived from the database the repository's own compose file
loads? Not "does a model get it right" — can a person, with the data in front
of them, write down the answer and show the query. Each derivation is recorded
with the executable selector that produced it.

disposition	n	what it means
derived	14	the answer follows from the loaded data
id_mismatch	12	identifiers in the question match none in the data
entity_absent	10	the question names equipment the data does not contain
input_file_absent	8	a file the question depends on is not in the repository
ambiguous_source	6	two shipped sources answer the same question differently
outside_answer_contract	3	the expected answer shape cannot be expressed unambiguously
window_absent	2	the time window asked about is not in the series
	55

41 of 55 cannot be graded against the shipped environment as it stands.

The largest category is identifiers, not missing data

This is the part we did not expect, and it is the most fixable.

All twelve Workorder scenarios are id_mismatch — ids 400–408, 410, 413 and
414; there is no 409, 411 or 412 in the set. The
shipped work-order table, src/couchdb/scenarios_data/shared/work_order/workorders.csv,
contains three rows, whose assetnum values are:

AHU2, CHILLER6, PUMP3

Scenario 400 asks "Get the work order of equipment CWC04013 for year 2017."
CWC04013 does not appear anywhere in that file. The guideline documents an
expected answer for this scenario at lines 120, 127 and 162, so it is one of the
scenarios whose ground truth is written down — and it is not answerable from
what ships.

Across all twelve, only two distinct pieces of equipment are named:

identifier	scenarios	period asked about
CWC04013	400, 401, 402, 403	2017
CWC04009	404–408, 410, 413, 414	2018–2021

Neither appears in the shipped table.

One identifier detail worth a look on its own, because it would defeat a
cross-source join even where the asset is present in both:

collection	spelling
iot/asset_profile_sample.json	Chiller 6
work_order/workorders.csv	CHILLER6

Same asset, two shipped collections, two spellings. PUMP3 and AHU2 agree
across both; Chiller 6 does not agree with itself.

The one we would raise first

ambiguous_source — scenarios 201, 207, 208, 211, 212, 215. These are
answerable twice, differently, depending on which shipped source you read.

We flag these ahead of the larger categories because absent data fails loudly
and this fails silently. A question whose data is missing produces an obvious
error. Two sources disagreeing produces a confident wrong grade, and no
automated scorer can see it happening. We deliberately did not pick a winner
between the two sources for any of the six — that is a call for whoever owns the
data, not for us.

What we are not claiming

Not that these 41 are unusable. Many would become gradeable with an
identifier fix or a missing file. The work-order set in particular turns on very
little: as the table above shows, the twelve scenarios between them name two
pieces of equipment, so what is absent is work-order history for two assets
rather than anything structural. We have not tried to say how many rows that
is, because it depends on what each question expects back, and we would rather
give you the identifier count we derived than a row count we guessed. We are
claiming only that as shipped they cannot be scored, and that 41/55 is large
enough to matter to anyone reporting an aggregate over this set.

Nor that the counts are beyond dispute. They are a judgement per question,
made once, by us. The dispositions are the arguable part; the identifier facts
above are not.

Verify the identifier claims directly

import csv, json
wo = list(csv.DictReader(open("src/couchdb/scenarios_data/shared/work_order/workorders.csv")))
print(len(wo), sorted({r["assetnum"] for r in wo}))
# -> 3 ['AHU2', 'CHILLER6', 'PUMP3']

a = json.load(open("src/couchdb/scenarios_data/shared/iot/asset_profile_sample.json"))
print(sorted(r["assetnum"] for r in a))
# -> ['AHU2', 'Chiller 6', 'Motor_01', 'PUMP3', 'hyd_1', 'mp_1']

What we are asking for

Nothing in the code. The same one line #506 asks for — whether the public
artefacts are meant to be scoreable end-to-end without the internal ground-truth
dataset — plus, if it is useful to you, the disposition table above in
docs/evaluation.md so the next external user starts from the number instead of
deriving it.

Happy to hand over the full audit — 55 rows with the selector used for each
derivation — in whatever form is useful, or to rerun it against a corrected
manifest if identifiers get fixed.

Relationship to the other issues

This is the second of four issues that a single documentation PR closes,
after #506. The other two are distinct things an external user trips over,
sharing a fix location rather than a root cause: what the MCQA pools do and do
not exercise, and which questions are answerable without touching the tools.
Both of those hold whatever the answer to #506 turns out to be.

The content is written; its location is your call — four smaller PRs, a
different file, or this shape.

---

Findings 1–6 reached the maintainers by email on 10 August. This continues that
exchange, and is the third of the six issues announced in #503.