Skip to content

docs(lessons): add lessons 86 and 87 on blind test scaffolding - #1129

Open
dekobon wants to merge 1 commit into
mainfrom
docs/lessons-86-87
Open

docs(lessons): add lessons 86 and 87 on blind test scaffolding#1129
dekobon wants to merge 1 commit into
mainfrom
docs/lessons-86-87

Conversation

@dekobon

@dekobon dekobon commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Adds two lessons and one sub-example, from investigating #1091 and
#1054 — the two bug-labelled issues the last lessons pass left
uninvestigated.

Both turned out to be test-quality failures rather than the perf work
their titles suggest, and both describe ways a test suite can look
thorough while being structurally unable to see a defect.

Lesson 86 — a normalizing test helper blinds every caller at once

check_ops sorted operators_str and operands_str — the values
returned by the code under test — alongside the expectations, from the
helper's first commit. Ops::operators and Ops::operands were built
from HashMap keys and Rust's RandomState reseeds per map instance,
so the order differed between two parses in the same process.

The consequence reached users: bca ops printed byte-different output
for an unchanged input on consecutive runs. Eighteen per-language
tests parsed real source and compared real vocabularies, and not one
could observe it. The fix sorts in production and removes only the
actual-side sort, which turned those same eighteen callers into
ordering regression tests without writing one.

Lesson 87 — an assertion can be correct and still be about the wrong rows

last_emitted_metric_group_uses_closing_connector selected group lines
at three columns while metric groups sit six columns in. It matched
exactly one line — the metrics header — and its closing-connector
assertion was a true statement about that header. The test passed with
every metric group rendering a dangling |-, which is the precise
defect its name claims to prevent.

The same fix records why contains cannot work on this output at all:
a deeper rail ends with the shallower one, and sum / value recur
across groups, so a substring search accepts exactly the
mis-indentation under test.

Lesson 85 — third sub-example

cyclomatic.modified is the only nested metric object, so
dump_object's per-field truncate is the only thing keeping the fields
that follow a nested object on the right rail. Removing it mis-indents
cyclomatic's trailing sum / value in every dump — with all 2,869
lib tests still passing. An independent instance of lesson 85's thesis
from a different subsystem.

Verification

Reviewed before commit, which caught two false claims in the first
draft — both now fixed and re-verified against the tree:

Six further imprecisions were corrected: the ambiguous sort_unstable
referent, the unreproducible 27 denominator (now scoped to the tests
under ops and output::dump_ops as they stood then), an overstated
"deletes a property from the contract" (the order was documented as
arbitrary, so the helper accommodated a sanctioned non-guarantee), an
overstated equivalence with lesson 23, the undated 2,869, and a
truncate description that implied a dedicated line.

Every figure in both entries was checked against the repository rather
than transcribed from the issue bodies. Entries cite issues and PRs,
never commit hashes — this repository rebase-merges, so branch hashes
are orphaned on landing. rumdl check passes; the change is purely
additive (two new entries appended, one sub-example inserted), so no
existing lesson was renumbered and the number-based references in
.claude/skills/ and .claude/rules/ are unaffected.

Two entries from the #1091 and #1054 fixes, plus a third sub-example
for lesson 85 from the same investigation.

Lesson 86: a test helper that canonicalises the value under test
deletes that dimension from every caller at once. `check_ops` sorted
`operators_str` and `operands_str` alongside the expectations from its
first commit, so eighteen per-language tests could not observe that
`Ops`'s vocabularies came off `HashMap` keys in per-instance-random
order — while `bca ops` printed byte-different output on consecutive
runs. Removing only the actual-side sort turned those same callers
into ordering regression tests for free.

Lesson 87: an assertion gets the review attention, but the selector
decides what it is about. A connector test filtered group lines at
three columns where groups sit six columns in, so it matched only the
`metrics` header and asserted something true about that instead — it
passed with every metric group rendering a dangling `|-`. Substring
matching cannot work here at all, because a deeper rail ends with the
shallower one.

Every figure was verified against the tree rather than transcribed
from the issues: the caller count is eighteen, not the "~20" issue
#1091 records, and the 20 of 27 failing tests decomposes as those
eighteen plus the two tests the fix added. Cites issues and PRs, never
commit hashes, which this repository's rebase merges orphan.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant