diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 4094f81317..50f036aa4f 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -222,10 +222,30 @@ jobs: if: ${{ !cancelled() }} run: python3 scripts/gap_snapshot.py --self-test + # Two halves of the same gate (#7582). + # + # `--self-test` unit-checks the checker. `--audit` runs the checker's + # OFFLINE half against the committed files: every known_failures.json + # entry must carry provenance (issue + date, #797), name a test that + # still exists, and — for gap-suite entries — be corroborated by + # `test-parity/gap_snapshot.json`, which is GENERATED and bidirectional. + # An entry absent from that snapshot is one the snapshot asserts passes, + # i.e. a suppression that has outlived its bug. + # + # This belongs on `lint` rather than only on `parity` because `parity` is + # TAG-gated: the live half of the ratchet fires after every merge it was + # meant to judge. `test_gap_diagchannel_3082_3084_3085_3086` sat here from + # 2026-07-04 and absorbed a real data-loss regression for six days (#7580); + # this step would have named it on the day it was added. Costs ~0.1s and + # runs no tests. - name: Platform-aware parity allowlist self-test if: ${{ !cancelled() }} run: python3 scripts/parity_known_failures.py --self-test + - name: Parity allowlist ratchet (provenance + stale entries) + if: ${{ !cancelled() }} + run: python3 scripts/parity_known_failures.py --audit + # Moving-GC gate wiring. The GC gates are the ones this repo has most # often found unable to fail (CLAUDE.md's four hazards), and every miss so # far was caught by a human re-deriving it mid-incident. This asserts the @@ -1739,7 +1759,12 @@ jobs: cat test-parity/reports/parity_threshold_latest.md >> "$GITHUB_STEP_SUMMARY" exit "$status" - - name: Check for new failures + # Bidirectional since #7582: red on a failure that is not allowed here, + # AND red on an allowlist entry whose test ran on this platform and + # PASSED. The second direction is the one that matters — the file was a + # pure suppression list, so a fixed test kept its entry forever and the + # entry silently absorbed that test's next regression (#7580). + - name: Check for new and stale failures run: >- python3 scripts/parity_known_failures.py --report test-parity/reports/latest.json diff --git a/CLAUDE.md b/CLAUDE.md index 8a9671604a..2d35fd6fff 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,14 +8,14 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Perry is a native TypeScript compiler written in Rust that compiles TypeScript source code directly to native executables. It uses SWC for TypeScript parsing and LLVM for code generation. -**Current Version:** 0.5.1344 +**Current Version:** 0.5.1345 ## TypeScript Parity Status Tracked via the gap test suite (`test-files/test_gap_*.ts`). Compared byte-for-byte against `node --experimental-strip-types`. Run via `./scripts/run_gap_tests.sh` (a thin wrapper over `run_parity_tests.sh --filter test_gap_` that builds the compiler itself and gates on no new untriaged failures). -**The oracle is Node `26.5.0`, pinned in `.node-version` at the repo root** — the single source of truth every CI workflow reads via `setup-node`'s `node-version-file`. **Run the gap suite against that exact version locally**, or your results won't match CI. The version is a *correctness input*, not an incidental toolchain detail: when node can't run a test (a feature newer than the pinned node), node exits non-zero, the harness classifies it `node_fail`, and the test is **silently dropped from the gate** rather than going red. CI sat on Node 22 while the suite grew Node 24/26 features, which hid 14 tests — all of Temporal, plus DisposableStack, Float16Array, and `Uint8Array` base64/hex (#6364). Node patch releases also change observable output (error-message text, `v8` heap fields), which is why the pin is exact. Raising it is a deliberate act: measure the failure delta under both oracles first, then triage what it exposes. +**The oracle is Node `26.5.1`, pinned in `.node-version` at the repo root** (the FILE is authoritative — this prose drifted once, #7599) — the single source of truth every CI workflow reads via `setup-node`'s `node-version-file`. **Run the gap suite against that exact version locally**, or your results won't match CI. The version is a *correctness input*, not an incidental toolchain detail: when node can't run a test (a feature newer than the pinned node), node exits non-zero, the harness classifies it `node_fail`, and the test is **silently dropped from the gate** rather than going red. CI sat on Node 22 while the suite grew Node 24/26 features, which hid 14 tests — all of Temporal, plus DisposableStack, Float16Array, and `Uint8Array` base64/hex (#6364). Node patch releases also change observable output (error-message text, `v8` heap fields), which is why the pin is exact. Raising it is a deliberate act: measure the failure delta under both oracles first, then triage what it exposes. Two workflows are deliberately exempt and say so inline: `node-core-subset.yml` derives its Node from `test-compat/node-core/pinned-version.txt` (it runs Node's *own* test corpus, which must match its own Node line), and the two release workflows use Node purely as an npm *publishing* toolchain. diff --git a/Cargo.lock b/Cargo.lock index bfd20135f1..b33c920ede 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5547,7 +5547,7 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" [[package]] name = "perry" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "base64", @@ -5607,14 +5607,14 @@ dependencies = [ [[package]] name = "perry-api-manifest" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "serde", ] [[package]] name = "perry-audio-miniaudio" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "cc", "libc", @@ -5622,7 +5622,7 @@ dependencies = [ [[package]] name = "perry-codegen" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "inkwell", @@ -5639,7 +5639,7 @@ dependencies = [ [[package]] name = "perry-codegen-arkts" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "perry-hir", @@ -5647,7 +5647,7 @@ dependencies = [ [[package]] name = "perry-codegen-glance" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "perry-hir", @@ -5655,7 +5655,7 @@ dependencies = [ [[package]] name = "perry-codegen-js" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "perry-dispatch", @@ -5664,7 +5664,7 @@ dependencies = [ [[package]] name = "perry-codegen-swiftui" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "perry-hir", @@ -5672,7 +5672,7 @@ dependencies = [ [[package]] name = "perry-codegen-wasm" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "base64", @@ -5684,7 +5684,7 @@ dependencies = [ [[package]] name = "perry-codegen-wear-tiles" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "perry-hir", @@ -5692,7 +5692,7 @@ dependencies = [ [[package]] name = "perry-container-compose" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "async-trait", @@ -5721,14 +5721,14 @@ dependencies = [ [[package]] name = "perry-container-e2e" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", ] [[package]] name = "perry-diagnostics" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "serde", "serde_json", @@ -5736,7 +5736,7 @@ dependencies = [ [[package]] name = "perry-dispatch" -version = "0.5.1344" +version = "0.5.1345" [[package]] name = "perry-doc-fixture-my-bindings" @@ -5747,7 +5747,7 @@ dependencies = [ [[package]] name = "perry-doc-tests" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "clap", @@ -5762,7 +5762,7 @@ dependencies = [ [[package]] name = "perry-ext-ads" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "block2", "objc2", @@ -5772,7 +5772,7 @@ dependencies = [ [[package]] name = "perry-ext-argon2" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "argon2", "perry-ffi", @@ -5780,7 +5780,7 @@ dependencies = [ [[package]] name = "perry-ext-axios" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "reqwest", @@ -5789,7 +5789,7 @@ dependencies = [ [[package]] name = "perry-ext-bcrypt" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "bcrypt", "perry-ffi", @@ -5797,7 +5797,7 @@ dependencies = [ [[package]] name = "perry-ext-better-sqlite3" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "rusqlite", @@ -5805,7 +5805,7 @@ dependencies = [ [[package]] name = "perry-ext-cheerio" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "scraper", @@ -5813,7 +5813,7 @@ dependencies = [ [[package]] name = "perry-ext-commander" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "perry-runtime", @@ -5821,7 +5821,7 @@ dependencies = [ [[package]] name = "perry-ext-cron" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "chrono", "cron", @@ -5831,7 +5831,7 @@ dependencies = [ [[package]] name = "perry-ext-dayjs" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "chrono", "perry-ffi", @@ -5839,7 +5839,7 @@ dependencies = [ [[package]] name = "perry-ext-decimal" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "rust_decimal", @@ -5847,7 +5847,7 @@ dependencies = [ [[package]] name = "perry-ext-dotenv" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "serde_json", @@ -5855,7 +5855,7 @@ dependencies = [ [[package]] name = "perry-ext-ethers" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "rand 0.10.1", @@ -5863,7 +5863,7 @@ dependencies = [ [[package]] name = "perry-ext-events" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "perry-runtime", @@ -5871,14 +5871,14 @@ dependencies = [ [[package]] name = "perry-ext-exponential-backoff" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", ] [[package]] name = "perry-ext-fastify" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "bytes", "http-body-util", @@ -5896,7 +5896,7 @@ dependencies = [ [[package]] name = "perry-ext-fetch" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "bytes", "lazy_static", @@ -5909,7 +5909,7 @@ dependencies = [ [[package]] name = "perry-ext-http" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "bytes", "h2", @@ -5933,7 +5933,7 @@ dependencies = [ [[package]] name = "perry-ext-ioredis" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "lazy_static", "perry-ffi", @@ -5943,7 +5943,7 @@ dependencies = [ [[package]] name = "perry-ext-jsonwebtoken" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "base64", "jsonwebtoken", @@ -5954,7 +5954,7 @@ dependencies = [ [[package]] name = "perry-ext-lru-cache" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "lru", "perry-ffi", @@ -5963,7 +5963,7 @@ dependencies = [ [[package]] name = "perry-ext-moment" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "chrono", "perry-ffi", @@ -5971,7 +5971,7 @@ dependencies = [ [[package]] name = "perry-ext-mongodb" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "bson", "futures-util", @@ -5983,7 +5983,7 @@ dependencies = [ [[package]] name = "perry-ext-mysql2" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "chrono", "perry-ffi", @@ -5993,7 +5993,7 @@ dependencies = [ [[package]] name = "perry-ext-nanoid" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "nanoid", "perry-ffi", @@ -6002,7 +6002,7 @@ dependencies = [ [[package]] name = "perry-ext-net" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "bytes", "perry-ffi", @@ -6015,7 +6015,7 @@ dependencies = [ [[package]] name = "perry-ext-node-forge" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "const-oid 0.9.6", "der 0.7.10", @@ -6034,7 +6034,7 @@ dependencies = [ [[package]] name = "perry-ext-nodemailer" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "lettre", "perry-ffi", @@ -6044,7 +6044,7 @@ dependencies = [ [[package]] name = "perry-ext-pdf" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "printpdf", @@ -6052,7 +6052,7 @@ dependencies = [ [[package]] name = "perry-ext-pg" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "sqlx", @@ -6061,7 +6061,7 @@ dependencies = [ [[package]] name = "perry-ext-ratelimit" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "governor", "perry-ffi", @@ -6069,7 +6069,7 @@ dependencies = [ [[package]] name = "perry-ext-sharp" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "fast_image_resize", "image", @@ -6079,14 +6079,14 @@ dependencies = [ [[package]] name = "perry-ext-slugify" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", ] [[package]] name = "perry-ext-streams" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "lazy_static", "perry-ffi", @@ -6095,7 +6095,7 @@ dependencies = [ [[package]] name = "perry-ext-undici" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "perry-runtime", @@ -6104,7 +6104,7 @@ dependencies = [ [[package]] name = "perry-ext-uuid" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "uuid", @@ -6112,7 +6112,7 @@ dependencies = [ [[package]] name = "perry-ext-validator" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ffi", "regex", @@ -6122,7 +6122,7 @@ dependencies = [ [[package]] name = "perry-ext-ws" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "futures-util", "lazy_static", @@ -6135,7 +6135,7 @@ dependencies = [ [[package]] name = "perry-ext-zlib" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "brotli", "flate2", @@ -6145,7 +6145,7 @@ dependencies = [ [[package]] name = "perry-ffi" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "dashmap", "once_cell", @@ -6154,7 +6154,7 @@ dependencies = [ [[package]] name = "perry-hir" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "perry-api-manifest", @@ -6172,7 +6172,7 @@ dependencies = [ [[package]] name = "perry-parser" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "perry-diagnostics", @@ -6184,7 +6184,7 @@ dependencies = [ [[package]] name = "perry-runtime" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "base64", @@ -6226,14 +6226,14 @@ dependencies = [ [[package]] name = "perry-runtime-static" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-runtime", ] [[package]] name = "perry-stdlib" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "aes 0.8.4", "aes 0.9.1", @@ -6328,14 +6328,14 @@ dependencies = [ [[package]] name = "perry-stdlib-static" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-stdlib", ] [[package]] name = "perry-transform" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "perry-hir", @@ -6344,14 +6344,14 @@ dependencies = [ [[package]] name = "perry-ui" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ui-model", ] [[package]] name = "perry-ui-android" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "base64", "itoa", @@ -6368,7 +6368,7 @@ dependencies = [ [[package]] name = "perry-ui-geisterhand" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "rand 0.10.1", "serde", @@ -6378,7 +6378,7 @@ dependencies = [ [[package]] name = "perry-ui-gtk4" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "base64", "cairo-rs 0.22.0", @@ -6401,7 +6401,7 @@ dependencies = [ [[package]] name = "perry-ui-ios" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "base64", "block2", @@ -6417,7 +6417,7 @@ dependencies = [ [[package]] name = "perry-ui-macos" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "base64", "block2", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "perry-ui-model" -version = "0.5.1344" +version = "0.5.1345" [[package]] name = "perry-ui-test" @@ -6443,11 +6443,11 @@ dependencies = [ [[package]] name = "perry-ui-testkit" -version = "0.5.1344" +version = "0.5.1345" [[package]] name = "perry-ui-tvos" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "base64", "block2", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "perry-ui-visionos" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "base64", "block2", @@ -6479,7 +6479,7 @@ dependencies = [ [[package]] name = "perry-ui-watchos" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "block2", "libc", @@ -6492,7 +6492,7 @@ dependencies = [ [[package]] name = "perry-ui-windows" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "base64", "libc", @@ -6509,14 +6509,14 @@ dependencies = [ [[package]] name = "perry-ui-windows-winui" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "perry-ui-windows", ] [[package]] name = "perry-updater" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "anyhow", "base64", @@ -6532,7 +6532,7 @@ dependencies = [ [[package]] name = "perry-wasm-host" -version = "0.5.1344" +version = "0.5.1345" dependencies = [ "wasmi", ] diff --git a/Cargo.toml b/Cargo.toml index aab312087e..b30b10ed88 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -315,7 +315,7 @@ codegen-units = 16 codegen-units = 16 [workspace.package] -version = "0.5.1344" +version = "0.5.1345" edition = "2021" license = "MIT" repository = "https://github.com/PerryTS/perry" diff --git a/changelog.d/7599-known-failures-ratchet.md b/changelog.d/7599-known-failures-ratchet.md new file mode 100644 index 0000000000..80c6a48312 --- /dev/null +++ b/changelog.d/7599-known-failures-ratchet.md @@ -0,0 +1,14 @@ +**`test-parity/known_failures.json` is a ratchet now, not a suppression list (#7582; completes the audit #797 asked for on this file).** + +`scripts/parity_known_failures.py` computed `failures - allowed` and stopped, so an entry whose test had started passing was inert forever — it never failed, never reported, and never asked to be removed, while silently converting that test's next regression into a non-event. That is a fifth way a gate can be unable to fail alongside CLAUDE.md's four, and the worst of them: the job is genuinely green *and* genuinely running. `test_gap_diagchannel_3082_3084_3085_3086` was listed on 2026-07-04 for a feature cluster that had shipped; when #7105 broke the test again for an unrelated reason the entry absorbed it, and the resulting data-loss defect — every `let`/`const` in an ES module's top-level bare block reading back stale — sat six days until #7580. + +The check now runs both directions. A failure not allowed for the running platform is red, and so is **an allowlist entry whose test ran on that platform and passed**, with a message naming the entry to delete. "Now passes" requires positive evidence from the report's `results[]`: filtered out, absent from this shard, `node_fail`, `skipped`, or scoped elsewhere via `platforms` is never flagged, because absence of evidence is not a pass (#6364). `results[]` is mandatory rather than best-effort so the gate cannot silently degrade back to suppression-only, and a passing run prints how many entries it actually adjudicated. + +The `parity` job that consumes the file is tag-gated, so its live half fires after the merges it was meant to judge. `--audit` is the offline half and runs from `lint`, a required per-PR context, with no parity run at all: it enforces provenance (issue + date + a category from the documented enum — `toolchain` had slipped in unvalidated), rejects an entry naming a test file that no longer exists, and cross-checks every `test_gap_*` entry against the generated, bidirectional `gap_snapshot.json`. An entry absent from that snapshot is one the snapshot asserts passes; the check would have named the diagchannel entry on the day it was added. + +**The flip, in the same PR, because a new gate has never been green.** 37 entries → 10. The 27 retired are gap-suite entries the generated Linux `gap_snapshot.json` asserts pass — Linux is the platform this file's consumer runs on, that baseline was regenerated 2026-08-06, and the required `conformance-smoke` job enforces it bidirectionally on every PR. All 27 were also confirmed passing live on macOS (500-test gap run, plus a second auto-optimize pass over the ten whose first verdict came from a run mode that could not link them). Four of them were the auto-opt cold-object-cache zlib link failure whose tracker #6847 was closed COMPLETED on 2026-07-30, fixed by #7021 — those entries outlived their bug by a week. They also carried `category: "toolchain"`, which is not in the documented enum and which nothing validated. + +**The audit found a live reproduction of a closed issue.** `test_gap_zlib_4917_level` looked stale from the Linux snapshot and was retired with the rest — then reproduced #6847's exact signature on macOS/arm64 in auto-optimize mode, 3 runs out of 3: `Undefined symbols: _js_zlib_deflate_raw_sync, _js_zlib_inflate_raw_sync`. Its two zlib siblings pass in the same mode, so it is specific to the raw-sync entry points. #6847 needs reopening for macOS. The entry is kept with `platforms: ["macos"]` rather than globally: Linux passes the test, so an unscoped entry would go on suppressing a passing test on the platform the gate runs on — the exact failure mode this change removes. Scoped, it suppresses nothing in CI, records something true, and the ratchet still governs it. + +**Provenance refresh on the 9 survivors.** Every tracking issue in this file except #6477 is CLOSED. The four survivors that are bug claims with a closed tracker are re-categorised `bug-stale` — the category the schema already defines for exactly this — with the closed-tracker fact leading the reason; the `module-inventory` and `ci-env` survivors keep their categories, because `bug-stale` would assert something false about them, and carry the same note. `issue` still points at the original tracker: that is its provenance, not a claim that it is open. Each needs a new tracking issue, which is now visible instead of buried. + diff --git a/scripts/parity_known_failures.py b/scripts/parity_known_failures.py index 60401522dd..a7efad0d97 100644 --- a/scripts/parity_known_failures.py +++ b/scripts/parity_known_failures.py @@ -1,10 +1,56 @@ #!/usr/bin/env python3 -"""Check a parity report against the platform-aware known-failure allowlist.""" +"""Ratchet for the platform-aware parity known-failure allowlist. + +`test-parity/known_failures.json` used to be a pure SUPPRESSION list: this +script computed `failures - allowed` and stopped there, so an entry whose test +had started passing was inert forever. It never failed, never reported, and +never asked to be removed. That is a fifth way a gate can be unable to fail +(CLAUDE.md names four) and the most insidious, because the job is genuinely +green AND genuinely running — while the entry silently converts any future +regression of that exact test into a non-event. + +It cost a real bug. `test_gap_diagchannel_3082_3084_3085_3086` was listed on +2026-07-04 for a feature cluster that shipped; the entry stayed. When #7105 +broke the test again for an unrelated reason (`PreallocateBoxes` shadowing a +module-level global), the suppression absorbed it. The defect emptied every +`let`/`const` in a top-level bare block of an ES module that a sibling +`function` declaration read — six days, found by accident (#7580). + +So the check now runs in BOTH directions, the way +`scripts/gap_root_dominance_allowlist.json` and `test-parity/gap_snapshot.json` +already do in this repo (#7582): + + * fails and is NOT allowed here -> regression. Fix it, or triage it. + * allowed here and now PASSES -> STALE. Delete the entry, in this PR. + +"Now passes" is a per-platform verdict about the platform the run executed on, +and it needs POSITIVE evidence: the test must appear in the report's `results[]` +with status `pass`. An entry whose test did not run at all — filtered out, not +in this shard, `node_fail`, skipped — is never flagged. Absence of evidence is +not passing; that is exactly the hole a Node-22 pin used to hide 14 tests in +(#6364). + +Two extra teeth, both cheap: + + * `--audit` needs no report and no test run. It validates provenance (#797: + every entry carries an issue and the date it was listed) and cross-checks + the gap-suite entries against the committed `gap_snapshot.json` — a + generated, bidirectional baseline. A `test_gap_*` entry ABSENT from that + snapshot is one the snapshot asserts passes, i.e. stale. This matters + because the `parity` job that consumes this file is TAG-gated: without an + offline check the ratchet would almost never fire before a merge. Run from + `lint`, which is a required context. It would have flagged the diagchannel + entry on the day it was added. + + * the report path prints how many entries it actually ADJUDICATED. A gate + must assert its subject was live, not merely that nothing threw. +""" from __future__ import annotations import argparse import json +import re import sys from pathlib import Path @@ -12,6 +58,8 @@ ROOT = Path(__file__).resolve().parent.parent DEFAULT_REPORT = ROOT / "test-parity" / "reports" / "latest.json" DEFAULT_KNOWN = ROOT / "test-parity" / "known_failures.json" +DEFAULT_GAP_SNAPSHOT = ROOT / "test-parity" / "gap_snapshot.json" +DEFAULT_TEST_DIR = ROOT / "test-files" PLATFORM_ALIASES = { "cygwin": "windows", "darwin": "macos", @@ -24,6 +72,27 @@ "windows": "windows", } PLATFORMS = frozenset(PLATFORM_ALIASES.values()) +PASS = "pass" +# Mirrors test-parity/README.md ("Category definitions"). An undocumented +# category is a schema error: `toolchain` slipped in unnoticed precisely +# because nothing validated this field. +CATEGORIES = frozenset( + { + "ci-env", + "module-inventory", + "bug-open", + "bug-stale", + "gap-categorical", + "gap-bisect", + "untriaged", + } +) +ISSUE_RE = re.compile(r"^[1-9][0-9]*$") +DATE_RE = re.compile(r"^[0-9]{4}-[0-9]{2}-[0-9]{2}$") +# The snapshot cross-check only speaks for the suite the snapshot covers. +GAP_PREFIX = "test_gap_" +# gap_snapshot.json is generated on Linux and is the baseline required CI uses. +GAP_SNAPSHOT_PLATFORM = "linux" def normalize_platform(value: str) -> str: @@ -52,54 +121,196 @@ def report_failures(report: dict) -> set[str]: return result -def known_for_platform(known: dict, platform: str) -> tuple[set[str], list[str]]: - selected: set[str] = set() +def report_statuses(report: dict) -> dict[str, str]: + """test id -> status, from `results[]`. + + `results[]` is REQUIRED, not best-effort. `failures{}` alone can only ever + say what failed; the stale half of the ratchet needs to know what RAN and + passed. Silently degrading to "no results, nothing is stale" would make + this gate unable to fail again, which is the whole bug being fixed. + """ + results = report.get("results") + if not isinstance(results, list) or not results: + raise ValueError( + "parity report must contain a non-empty results[] array — the " + "stale-entry check cannot tell 'passed' from 'never ran' without it" + ) + statuses: dict[str, str] = {} + for item in results: + if not isinstance(item, dict): + raise ValueError(f"malformed results[] entry: {item!r}") + test_id, status = item.get("id"), item.get("status") + if not isinstance(test_id, str) or not isinstance(status, str): + raise ValueError(f"malformed results[] entry: {item!r}") + statuses[test_id] = status + return statuses + + +def validate_entry(test_id: str, record: object) -> tuple[list[str], bool]: + """Schema + provenance rules for one entry (#797). + + Returns (problems, platforms_ok). The second value is separate rather than + sniffed out of the message text: a `platforms` value the checker could not + parse must not be trusted to scope the entry either way, and deciding that + by substring-matching an error string would misfire on a test id. + """ problems: list[str] = [] - for test_id, record in known.items(): - if test_id == "_schema": - continue - if not isinstance(record, dict): - problems.append(f"{test_id}: entry must be an object") - continue - if not isinstance(record.get("category"), str) or not record["category"]: - problems.append(f"{test_id}: category must be a non-empty string") - if not isinstance(record.get("reason"), str) or not record["reason"]: - problems.append(f"{test_id}: reason must be a non-empty string") + if not isinstance(record, dict): + return [f"{test_id}: entry must be an object"], False - platforms = record.get("platforms") - if platforms is None: - selected.add(test_id) - continue + category = record.get("category") + if not isinstance(category, str) or not category: + problems.append(f"{test_id}: category must be a non-empty string") + elif category not in CATEGORIES: + problems.append( + f"{test_id}: category {category!r} is not one of " + f"{', '.join(sorted(CATEGORIES))} (see test-parity/README.md)" + ) + if not isinstance(record.get("reason"), str) or not record["reason"]: + problems.append(f"{test_id}: reason must be a non-empty string") + + # #797: provenance is mandatory. An entry with no issue and no date cannot + # be revalidated by anyone but its author, and that is how the pre-audit + # bare-name format decayed into orphans. + issue = record.get("issue") + if not isinstance(issue, str) or not ISSUE_RE.match(issue): + problems.append( + f"{test_id}: issue must be a GitHub issue number as a string " + f'(e.g. "793"); got {issue!r}' + ) + added = record.get("added") + if not isinstance(added, str) or not DATE_RE.match(added): + problems.append(f"{test_id}: added must be an ISO date YYYY-MM-DD; got {added!r}") + + platforms_ok = True + platforms = record.get("platforms") + if platforms is not None: if ( not isinstance(platforms, list) or not platforms or not all(isinstance(item, str) for item in platforms) ): problems.append(f"{test_id}: platforms must be a non-empty string array") + platforms_ok = False + else: + unknown = sorted( + {item for item in platforms if item.strip().lower() not in PLATFORM_ALIASES} + ) + if unknown: + problems.append(f"{test_id}: unknown platforms: {', '.join(unknown)}") + platforms_ok = False + elif len({normalize_platform(item) for item in platforms}) != len(platforms): + problems.append(f"{test_id}: platforms must not contain duplicates") + platforms_ok = False + return problems, platforms_ok + + +def entry_applies(record: object, platform: str) -> bool: + """Does this entry claim to cover `platform`? + + An entry scoped to other platforms says nothing about this host, so it is + neither honoured as a suppression nor judged stale here. + """ + if not isinstance(record, dict): + return False + platforms = record.get("platforms") + if platforms is None: + return True + if not isinstance(platforms, list): + return False + return platform in { + normalize_platform(item) for item in platforms if isinstance(item, str) + } + + +def known_for_platform(known: dict, platform: str) -> tuple[set[str], list[str]]: + selected: set[str] = set() + problems: list[str] = [] + for test_id, record in known.items(): + if test_id == "_schema": continue - normalized = [normalize_platform(item) for item in platforms] - unknown = sorted( - {item for item in platforms if item.strip().lower() not in PLATFORM_ALIASES} - ) - if unknown: - problems.append(f"{test_id}: unknown platforms: {', '.join(unknown)}") - continue - if len(normalized) != len(set(normalized)): - problems.append(f"{test_id}: platforms must not contain duplicates") + entry_problems, platforms_ok = validate_entry(test_id, record) + problems.extend(entry_problems) + # A malformed `platforms` value cannot be trusted to scope the entry. + if not platforms_ok: continue - if platform in normalized: + if entry_applies(record, platform): selected.add(test_id) return selected, problems -def check(report: dict, known: dict, platform_override: str | None = None) -> tuple[str, list[str], list[str]]: +def stale_entries(allowed: set[str], statuses: dict[str, str]) -> list[str]: + """Allowed-here entries whose test RAN on this platform and PASSED.""" + return sorted(t for t in allowed if statuses.get(t) == PASS) + + +def check( + report: dict, known: dict, platform_override: str | None = None +) -> tuple[str, list[str], list[str], list[str], int]: platform_value = platform_override or report.get("platform") or sys.platform if not isinstance(platform_value, str): raise ValueError("report platform must be a string") platform = normalize_platform(platform_value) failures = report_failures(report) + statuses = report_statuses(report) allowed, schema_problems = known_for_platform(known, platform) - return platform, sorted(failures - allowed), schema_problems + stale = stale_entries(allowed, statuses) + adjudicated = sum(1 for test_id in allowed if test_id in statuses) + return platform, sorted(failures - allowed), stale, schema_problems, adjudicated + + +def audit( + known: dict, + snapshot_tests: dict | None, + test_exists=None, +) -> tuple[list[str], list[str]]: + """Offline half: provenance + cross-check against the gap snapshot. + + Returns (schema_problems, stale_problems). Needs no parity run, so it can + live on a required per-PR job while the suite that consumes this file is + tag-gated. + """ + if test_exists is None: + def test_exists(test_id: str) -> bool: + return (DEFAULT_TEST_DIR / f"{test_id}.ts").exists() + + _, schema_problems = known_for_platform(known, GAP_SNAPSHOT_PLATFORM) + stale: list[str] = [] + for test_id, record in sorted(known.items()): + if test_id == "_schema": + continue + if not test_exists(test_id): + stale.append( + f"{test_id}: no test-files/{test_id}.ts — the test this entry " + f"suppresses no longer exists" + ) + continue + if snapshot_tests is None or not test_id.startswith(GAP_PREFIX): + continue + if not entry_applies(record, GAP_SNAPSHOT_PLATFORM): + continue + if test_id not in snapshot_tests: + stale.append( + f"{test_id}: test-parity/gap_snapshot.json (the generated " + f"{GAP_SNAPSHOT_PLATFORM} baseline) says this test PASSES" + ) + return schema_problems, stale + + +def print_stale(stale: list[str], platform: str | None = None) -> None: + where = f" on {platform}" if platform else "" + print( + f"\nSTALE known_failures.json entries — these tests PASS{where}:", + file=sys.stderr, + ) + for item in stale: + print(f" - {item}", file=sys.stderr) + print( + "\nDelete each one from test-parity/known_failures.json, in the SAME PR.\n" + "An entry that outlives its bug is not a triage note, it is a permanent\n" + "suppression of that test's next regression (#7582).", + file=sys.stderr, + ) def self_test() -> int: @@ -110,56 +321,173 @@ def self_test() -> int: "compile": [""], "crash": ["windows_crash"], }, + "results": [ + {"id": "all_hosts", "status": "parity_fail"}, + {"id": "windows_only", "status": "parity_fail"}, + {"id": "linux_only", "status": "parity_fail"}, + {"id": "windows_crash", "status": "crash"}, + ], } + + def entry(**overrides) -> dict: + base = { + "issue": "793", + "added": "2026-05-15", + "category": "bug-open", + "reason": "why", + } + base.update(overrides) + return base + known = { "_schema": {}, - "all_hosts": {"category": "bug-open", "reason": "all"}, - "windows_only": { - "category": "bug-open", - "reason": "win", - "platforms": ["win32"], - }, - "windows_crash": { - "category": "bug-open", - "reason": "crash", - "platforms": ["msys"], - }, - "linux_only": { - "category": "bug-open", - "reason": "linux", - "platforms": ["linux"], - }, + "all_hosts": entry(), + "windows_only": entry(platforms=["win32"]), + "windows_crash": entry(platforms=["msys"]), + "linux_only": entry(platforms=["linux"]), } - platform, new, problems = check(report, known) + platform, new, stale, problems, adjudicated = check(report, known) assert platform == "windows" assert new == ["linux_only"] + assert stale == [] assert problems == [] + # linux_only is not selected on windows, so it is not adjudicated here. + assert adjudicated == 3, adjudicated assert normalize_platform("darwin") == "macos" + # --- the ratchet: an allowed entry whose test now PASSES is stale -------- + fixed = dict(report) + fixed["failures"] = {"parity": ["linux_only"], "compile": [], "crash": []} + fixed["results"] = [ + {"id": "all_hosts", "status": PASS}, + {"id": "windows_only", "status": "parity_fail"}, + {"id": "windows_crash", "status": "crash"}, + {"id": "linux_only", "status": "parity_fail"}, + ] + _, _, stale, _, _ = check(fixed, known) + assert stale == ["all_hosts"], stale + + # A test that did NOT run is never stale — absence of evidence is not a pass. + absent = dict(fixed) + absent["results"] = [{"id": "windows_only", "status": "parity_fail"}] + _, _, stale, _, adjudicated = check(absent, known) + assert stale == [], stale + assert adjudicated == 1, adjudicated + + # node_fail / skipped are not passes either. + for status in ("node_fail", "skipped", "compile_fail"): + probe = dict(fixed) + probe["results"] = [{"id": "all_hosts", "status": status}] + _, _, stale, _, _ = check(probe, known) + assert stale == [], (status, stale) + + # An entry scoped to another platform is not judged here, even if it passes. + other = dict(fixed) + other["results"] = [{"id": "linux_only", "status": PASS}] + _, _, stale, _, _ = check(other, known) + assert stale == [], stale + + # results[] is mandatory: no silent degradation to a suppression-only gate. + try: + check({"platform": "linux", "failures": {"parity": [], "compile": [], "crash": []}}, known) + except ValueError as error: + assert "results[]" in str(error), error + else: # pragma: no cover - guarded by the assert below + raise AssertionError("a report with no results[] must be rejected") + + # --- schema / provenance ------------------------------------------------- malformed = { - "bad": {"category": "", "reason": "", "platforms": ["plan9"]}, + "bad": {"category": "", "reason": "", "platforms": ["plan9"], "issue": "", "added": "x"}, + } + _, _, _, problems, _ = check(report, malformed) + assert len(problems) == 5, problems + undocumented = {"bad": entry(category="toolchain")} + _, _, _, problems, _ = check(report, undocumented) + assert len(problems) == 1 and "not one of" in problems[0], problems + for field, value in (("issue", None), ("issue", 793), ("added", "2026-5-15")): + _, _, _, problems, _ = check(report, {"bad": entry(**{field: value})}) + assert len(problems) == 1 and field in problems[0], (field, value, problems) + + # --- offline audit ------------------------------------------------------- + gap_known = { + "test_gap_still_broken": entry(), + "test_gap_fixed": entry(), + "test_gap_linux_scoped_elsewhere": entry(platforms=["windows"]), + "test_parity_other_suite": entry(), } - _, _, problems = check(report, malformed) - assert len(problems) == 3, problems + snapshot = {"test_gap_still_broken": {"status": "parity_fail"}} + problems, stale = audit(gap_known, snapshot, test_exists=lambda _t: True) + assert problems == [], problems + assert len(stale) == 1 and stale[0].startswith("test_gap_fixed:"), stale + # A vanished test file is stale whatever the snapshot says. + _, stale = audit( + {"test_gap_gone": entry()}, snapshot, test_exists=lambda _t: False + ) + assert len(stale) == 1 and "no longer exists" in stale[0], stale + # No snapshot -> the file-existence half still runs, the cross-check does not. + _, stale = audit(gap_known, None, test_exists=lambda _t: True) + assert stale == [], stale + print("parity_known_failures self-test OK") return 0 +def run_audit(args: argparse.Namespace) -> int: + try: + known = load_json(args.known) if args.known.exists() else {} + snapshot_tests = None + if args.gap_snapshot.exists(): + snapshot_tests = load_json(args.gap_snapshot).get("tests", {}) + problems, stale = audit(known, snapshot_tests) + except (OSError, ValueError, json.JSONDecodeError) as error: + print(f"known-failure audit error: {error}", file=sys.stderr) + return 2 + + if problems: + print("Malformed known_failures.json entries:", file=sys.stderr) + for problem in problems: + print(f" - {problem}", file=sys.stderr) + return 2 + if stale: + print_stale(stale) + return 1 + + entries = sum(1 for key in known if key != "_schema") + scope = ( + f" and cross-checked {sum(1 for k in known if k.startswith(GAP_PREFIX))} " + f"gap entries against {args.gap_snapshot.name}" + if snapshot_tests is not None + else " (no gap snapshot found — cross-check skipped)" + ) + print(f"known_failures.json audit OK — {entries} entries carry provenance{scope}.") + return 0 + + def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--report", type=Path, default=DEFAULT_REPORT) parser.add_argument("--known", type=Path, default=DEFAULT_KNOWN) + parser.add_argument("--gap-snapshot", type=Path, default=DEFAULT_GAP_SNAPSHOT) parser.add_argument("--platform") + parser.add_argument( + "--audit", + action="store_true", + help="offline provenance + gap-snapshot cross-check; needs no parity run", + ) parser.add_argument("--self-test", action="store_true") args = parser.parse_args(argv) if args.self_test: return self_test() + if args.audit: + return run_audit(args) try: report = load_json(args.report) known = load_json(args.known) if args.known.exists() else {} - platform, new_failures, schema_problems = check(report, known, args.platform) + platform, new_failures, stale, schema_problems, adjudicated = check( + report, known, args.platform + ) except (OSError, ValueError, json.JSONDecodeError) as error: print(f"known-failure check error: {error}", file=sys.stderr) return 2 @@ -169,6 +497,8 @@ def main(argv: list[str] | None = None) -> int: for problem in schema_problems: print(f" - {problem}", file=sys.stderr) return 2 + + status = 0 if new_failures: print( f"NEW FAILURES on {platform} (not allowed for this platform):", @@ -176,10 +506,20 @@ def main(argv: list[str] | None = None) -> int: ) for test_id in new_failures: print(f" - {test_id}", file=sys.stderr) - return 1 + status = 1 + if stale: + print_stale(stale, platform) + status = 1 + if status: + return status total = len(report_failures(report)) - print(f"All {total} failures are known/triaged for {platform}.") + allowed, _ = known_for_platform(known, platform) + print( + f"All {total} failures are known/triaged for {platform}; " + f"{adjudicated}/{len(allowed)} allowlist entries were adjudicated by this " + f"run (the rest did not run here)." + ) return 0 diff --git a/test-parity/README.md b/test-parity/README.md index 7d66397212..57ea605f35 100644 --- a/test-parity/README.md +++ b/test-parity/README.md @@ -42,11 +42,62 @@ correctness input, so regenerate the snapshot on the pinned Node, on Linux ## `known_failures.json` — the full parity suite (tag-gated) Hand-curated allowlist consumed by the `parity` job in `test.yml`, which runs -the whole `test-files/*.ts` suite on tags and opt-in PRs. Still one-directional: -it only catches new failures. Migrating it to a generated snapshot needs a -full-suite baseline from a tag run and is a follow-up; until then its gap-suite -entries are redundant with `gap_snapshot.json` and are kept only so the -tag-gated job keeps passing. +the whole `test-files/*.ts` suite on tags and opt-in PRs. Migrating it to a +generated snapshot needs a full-suite baseline from a tag run and is still a +follow-up; until then its gap-suite entries overlap `gap_snapshot.json` and are +kept so the tag-gated job keeps passing. + +It is **bidirectional** as of #7582. `scripts/parity_known_failures.py` fails on +a failure that is not allowed here, and equally on an **entry whose test ran on +this platform and passed** — naming the entry to delete. It used to compute +`failures - allowed` and stop, which made an entry whose test had been fixed +inert forever: it never failed, never reported, and never asked to be removed, +while silently converting that test's next regression into a non-event. That is +not hypothetical. `test_gap_diagchannel_3082_3084_3085_3086` was listed on +2026-07-04 for a shipped feature cluster; when #7105 broke the test again for an +unrelated reason the entry absorbed it, and the underlying defect — every +`let`/`const` in an ES module's top-level bare block reading back stale — sat +for six days (#7580). + +"Now passes" needs positive evidence and is decided **per platform**: the test +must appear in the report's `results[]` with status `pass` for the platform the +run executed on. An entry whose test did not run — filtered out, not in this +shard, `node_fail`, `skipped`, or scoped to another platform via `platforms` — +is never flagged. Absence of evidence is not a pass; that is exactly the hole a +Node-22 pin used to hide 14 tests in (#6364). The passing run prints how many +entries it actually adjudicated, so a vacuously green gate is visible. + +Because the `parity` job is tag-gated, the live half of that ratchet only fires +*after* the merges it was meant to judge. The offline half runs on `lint`, which +is a required per-PR context, and needs no parity run at all: + +```bash +python3 scripts/parity_known_failures.py --audit +``` + +It enforces provenance (below), fails on an entry naming a test file that no +longer exists, and cross-checks every `test_gap_*` entry against +`gap_snapshot.json` — which is generated and bidirectional, so a gap entry +absent from it is one the snapshot asserts passes. That check would have named +the diagchannel entry on the day it was added. + +One sharp edge, stated rather than special-cased: a `ci-env` entry is by +definition one that *passes locally*, so running the full suite by hand — on a +box that happens to have the fixture the entry says CI lacks — will report it as +stale. That is the check working, not misfiring: on that host the entry is not +load-bearing. The authoritative verdict is the CI run. If a `ci-env` failure is +genuinely confined to one OS, scope it with `platforms` and the local run stops +adjudicating it at all. + +### Provenance (#797) + +Every entry must carry `issue`, `added`, `category`, and `reason`; the checker +rejects the file otherwise. `issue` is a GitHub issue number as a string, +`added` an ISO `YYYY-MM-DD` date, and `category` one of the values in the table +below — an undocumented category is a schema error (`toolchain` slipped in +unvalidated and went unnoticed). When the tracking issue closes but the test +still fails, re-triage: mark the entry `bug-stale` and file a new tracking +issue. Entries apply on every host by default. A failure that is specific to one or more operating systems can add a `platforms` array containing `linux`, `macos`, diff --git a/test-parity/known_failures.json b/test-parity/known_failures.json index dc802ad606..65f574f6a3 100644 --- a/test-parity/known_failures.json +++ b/test-parity/known_failures.json @@ -1,235 +1,76 @@ { "_schema": { - "description": "Each entry maps a parity-suite test name to a structured record. The CI gate in .github/workflows/test.yml reads `keys[]` from this file — a test that fails AND is not a key here breaks CI. Add entries here only with full provenance (issue + added date) so we can revalidate.", + "description": "Each entry maps a parity-suite test name to a structured record. This is a RATCHET, not a suppression list (#7582): scripts/parity_known_failures.py fails on a failure that is not listed here for the running platform, AND on an entry whose test ran on that platform and PASSED — naming the entry to delete. Delete it in the same PR that fixes the test. An entry that outlives its bug is a permanent suppression of that test's next regression: test_gap_diagchannel_3082_3084_3085_3086 was listed on 2026-07-04 for a shipped feature cluster, absorbed an unrelated data-loss regression when #7105 landed, and hid it for six days (#7580).", "fields": { - "issue": "Open GitHub issue number tracking this failure (e.g. \"793\"), or null when the failure is environmental / pending triage. Closed issues must be re-evaluated — flag the entry as `category: bug-stale` until a new tracking issue is filed.", - "added": "ISO date (YYYY-MM-DD) when the test was first skip-listed. Use 2026-05-15 for entries inherited from the pre-audit format where the historical date is unknown.", - "category": "ci-env | module-inventory | bug-open | bug-stale | gap-categorical | gap-bisect — see test-parity/README.md for definitions.", - "reason": "Free-text explanation. Keep the most-actionable signal first — what changes the verdict (a fixture, a Perry fix, a Node spec change).", - "platforms": "Optional non-empty array of linux, macos, windows, or other. Omit when the failure is accepted on every platform." + "issue": "REQUIRED. GitHub issue number tracking this failure, as a string (e.g. \"793\"). Closed issues must be re-evaluated — flag the entry as `category: bug-stale` until a new tracking issue is filed. The checker rejects a missing/null/non-numeric value.", + "added": "REQUIRED. ISO date (YYYY-MM-DD) when the test was first skip-listed. Use 2026-05-15 for entries inherited from the pre-audit format where the historical date is unknown.", + "category": "REQUIRED, and validated against the enum: ci-env | module-inventory | bug-open | bug-stale | gap-categorical | gap-bisect | untriaged — see test-parity/README.md for definitions. An undocumented value is a schema error.", + "reason": "REQUIRED. Free-text explanation. Keep the most-actionable signal first — what changes the verdict (a fixture, a Perry fix, a Node spec change).", + "platforms": "Optional non-empty array of linux, macos, windows, or other. Omit when the failure is accepted on every platform. An entry scoped to other platforms is neither honoured nor judged stale on this host — 'now passes' is a per-platform verdict." }, - "scope": "Consumed by the TAG-GATED `parity` job (full test-files/*.ts suite) only. The per-PR gap-suite gate moved to test-parity/gap_snapshot.json, which is generated and checked in BOTH directions (see test-parity/README.md); the test_gap_* entries below are redundant with it and kept only so the tag-gated job keeps passing. Migrating this file to a generated snapshot needs a full-suite baseline from a tag run — follow-up to #797." + "scope": "Consumed by the TAG-GATED `parity` job (full test-files/*.ts suite). Because that job runs after the merges it would judge, the OFFLINE half of the ratchet runs on `lint` (a required per-PR context) via `python3 scripts/parity_known_failures.py --audit`: it enforces the provenance fields above, rejects an entry naming a test file that no longer exists, and cross-checks every test_gap_* entry against test-parity/gap_snapshot.json — a generated, bidirectional baseline, so a gap entry absent from it is one the snapshot asserts passes. Migrating this whole file to a generated snapshot needs a full-suite baseline from a tag run — follow-up to #797." }, "test_parity_stream": { "issue": "793", "added": "2026-05-15", "category": "module-inventory", - "reason": "Node.js module inventory — `node:stream` surface not fully implemented. Tracker for surface coverage; flips to PASS as each API lands. Not a regression." + "reason": "Tracking issue #793 (roadmap umbrella) is CLOSED — this needs a live surface tracker (audited 2026-08-07, #7582). Node.js module inventory — `node:stream` surface not fully implemented. Tracker for surface coverage; flips to PASS as each API lands. Not a regression." }, "test_parity_stream_web": { "issue": "793", "added": "2026-05-15", "category": "module-inventory", - "reason": "Node.js module inventory — `node:stream/web` (WHATWG streams) surface not fully implemented. Tracker for surface coverage; flips to PASS as each API lands. Not a regression." + "reason": "Tracking issue #793 (roadmap umbrella) is CLOSED — this needs a live surface tracker (audited 2026-08-07, #7582). Node.js module inventory — `node:stream/web` (WHATWG streams) surface not fully implemented. Tracker for surface coverage; flips to PASS as each API lands. Not a regression." }, "test_sock_write_map": { "issue": "1634", "added": "2026-05-15", "category": "ci-env", - "reason": "Tracked in #1634 (parity CI environment fixtures). Net test passes locally with an echo server running, fails on Linux CI (no echo fixture server). Environmental — issue #91 dispatch fix landed in v0.5.581 and is no longer the cause; needs a CI-side fixture." + "reason": "Tracking issue #1634 is CLOSED — this needs a live CI-fixture tracker (audited 2026-08-07, #7582; not adjudicated locally, the echo-server port was already bound). Tracked in #1634 (parity CI environment fixtures). Net test passes locally with an echo server running, fails on Linux CI (no echo fixture server). Environmental — issue #91 dispatch fix landed in v0.5.581 and is no longer the cause; needs a CI-side fixture." }, "test_ramda_sum": { "issue": "1634", "added": "2026-05-18", "category": "ci-env", - "reason": "Tracked in #1634 (parity CI environment fixtures). Ramda npm-package fixture failure on Linux CI; observed on #1038's CI run pre-merge. Companion to the existing test_ramda_user_import skip in the compile-smoke list — the parity harness can't npm-install ramda. File a CI-side fixture issue if/when this matters for sweep coverage." + "reason": "Tracking issue #1634 is CLOSED — this needs a live CI-fixture tracker (audited 2026-08-07, #7582). Tracked in #1634 (parity CI environment fixtures). Ramda npm-package fixture failure on Linux CI; observed on #1038's CI run pre-merge. Companion to the existing test_ramda_user_import skip in the compile-smoke list — the parity harness can't npm-install ramda. File a CI-side fixture issue if/when this matters for sweep coverage." }, "test_gap_2159_defineproperty_class_prototype": { "issue": "2159", "added": "2026-07-04", - "category": "bug-open", - "reason": "defineProperty on class prototypes — standing gap (issue encoded in name); confirmed standing (pre-dates v0.5.1205) in the #5917 parity diff." + "category": "bug-stale", + "reason": "RE-TRIAGE: tracking issue #2159 is CLOSED but this still fails (audited 2026-08-07, #7582) — needs a new issue. defineProperty on class prototypes; standing gap, confirmed pre-dating v0.5.1205 in the #5917 parity diff." }, "test_gap_2514_settracesigint": { "issue": "2514", "added": "2026-07-04", - "category": "bug-open", - "reason": "process SIGINT trace hook gap; standing per #5917 diff." - }, - "test_gap_2754_2907_2908_bigint_semantics": { - "issue": "2754", - "added": "2026-07-04", - "category": "bug-open", - "reason": "BigInt semantics cluster (#2754/#2907/#2908); standing per #5917 diff." - }, - "test_gap_3828_function_method_values": { - "issue": "3828", - "added": "2026-07-04", - "category": "bug-open", - "reason": "Function method value forms (#3828); standing per #5917 diff." - }, - "test_gap_class_expr_extends_static_call_this": { - "issue": "5917", - "added": "2026-07-04", - "category": "gap-bisect", - "reason": "class-expression extends + static call `this` — no dedicated issue; tracked via the #5917 standing-tail worklist (watch for newness: flagged there as one of two to eyeball)." - }, - "test_gap_class_expr_static_this": { - "issue": "5917", - "added": "2026-07-04", - "category": "gap-bisect", - "reason": "class-expression static `this` — tracked via #5917 standing-tail worklist (eyeball-for-newness pair)." - }, - "test_gap_console_bare_global": { - "issue": "5917", - "added": "2026-07-04", - "category": "gap-categorical", - "reason": "console formatting/global surface — CLAUDE.md categorical gap (console.dir/group formatting); standing per #5917." - }, - "test_gap_console_validate_write": { - "issue": "5917", - "added": "2026-07-04", - "category": "gap-categorical", - "reason": "console write validation — categorical console gap; standing per #5917." - }, - "test_gap_dyn_index_get_denormal_safe": { - "issue": "5917", - "added": "2026-07-04", - "category": "bug-open", - "reason": "dynamic index-get denormal safety; standing per #5917 diff." - }, - "test_gap_fetch_instanceof_5433": { - "issue": "5433", - "added": "2026-07-04", - "category": "bug-open", - "reason": "fetch Response/Request instanceof (#5433); standing per #5917." - }, - "test_gap_fetch_response_json_init": { - "issue": "5917", - "added": "2026-07-04", - "category": "bug-open", - "reason": "Response.json()/init surface; standing per #5917 diff." - }, - "test_gap_global_apis": { - "issue": "5917", - "added": "2026-07-04", - "category": "module-inventory", - "reason": "global API surface inventory; standing per #5917." - }, - "test_gap_module_const_local_shadow": { - "issue": "5917", - "added": "2026-07-04", - "category": "bug-open", - "reason": "module-const local shadowing; standing per #5917 diff." - }, - "test_gap_node_v8_3137plus": { - "issue": "3137", - "added": "2026-07-04", - "category": "module-inventory", - "reason": "node:v8 surface (#3137+); standing per #5917." - }, - "test_gap_object_methods": { - "issue": "5917", - "added": "2026-07-04", - "category": "bug-open", - "reason": "Object static-method tail; standing per #5917 (cf. #5588 lineage)." + "category": "bug-stale", + "reason": "RE-TRIAGE: tracking issue #2514 is CLOSED but this still fails (audited 2026-08-07, #7582) — needs a new issue. process SIGINT trace hook gap; standing per the #5917 diff." }, "test_gap_perfhooks_3088_3008_3010_3011": { "issue": "3088", "added": "2026-07-04", - "category": "module-inventory", - "reason": "perf_hooks cluster (#3088/#3008/#3010/#3011); standing per #5917." - }, - "test_gap_sqlite_3183plus": { - "issue": "3183", - "added": "2026-07-04", - "category": "module-inventory", - "reason": "node:sqlite surface (#3183+); standing per #5917." - }, - "test_gap_static_member_call": { - "issue": "5917", - "added": "2026-07-04", - "category": "bug-open", - "reason": "static member call form; standing per #5917 diff." - }, - "test_gap_string_coercion_tostring": { - "issue": "5917", - "added": "2026-07-04", - "category": "bug-open", - "reason": "string coercion/toString edge; standing per #5917 diff." - }, - "test_gap_string_locale_2781_2845_2897": { - "issue": "2781", - "added": "2026-07-04", - "category": "bug-open", - "reason": "locale string cluster (#2781/#2845/#2897); standing per #5917." - }, - "test_gap_symbols": { - "issue": "5917", - "added": "2026-07-04", - "category": "gap-categorical", - "reason": "Symbol surface tail; standing per #5917." + "category": "bug-stale", + "reason": "RE-TRIAGE: tracking issue #3088 is CLOSED but this still fails (audited 2026-08-07, #7582) — needs a new issue. node:perf_hooks cluster (#3088/#3008/#3010/#3011) module-inventory gap." }, "test_gap_stream_tee_tick_parity": { "issue": "6477", "added": "2026-07-20", "category": "bug-open", - "reason": "Web Streams tee cold-start fires one extra microtask tick before the first branch read on a pre-buffered+closed source (t2 vs Node's t1). Delicate cadence calibration (post-#6657 tee/pipe tick parity); a naive fix risks the Next.js Flight byte-parity #6657 tuned. Deferred to the streams pull-ordering work in #6477." + "reason": "Re-verified still failing 2026-08-07 (#7582 audit); #6477 is the one tracking issue in this file that is still OPEN. Web Streams tee cold-start fires one extra microtask tick before the first branch read on a pre-buffered+closed source (t2 vs Node's t1). Delicate cadence calibration (post-#6657 tee/pipe tick parity); a naive fix risks the Next.js Flight byte-parity #6657 tuned. Deferred to the streams pull-ordering work in #6477." }, "test_gap_v8_2_3680plus": { "issue": "3680", "added": "2026-07-04", - "category": "module-inventory", - "reason": "node:v8 second cluster (#3680+); standing per #5917." - }, - "test_gap_v8_heap_snapshot_4916": { - "issue": "4916", - "added": "2026-07-04", - "category": "bug-open", - "reason": "v8 heap snapshot (#4916); standing per #5917." - }, - "test_gap_yieldstar_inherited_iterator_this": { - "issue": "5917", - "added": "2026-07-04", - "category": "gap-bisect", - "reason": "yield* inherited-iterator `this` — tracked via #5917 standing-tail worklist (eyeball-for-newness pair)." - }, - "test_gap_zlib_3285_params": { - "issue": "3285", - "added": "2026-07-04", - "category": "bug-open", - "reason": "zlib params (#3285); standing per #5917." - }, - "test_gap_readline_3698plus": { - "issue": "3698", - "added": "2026-07-04", - "category": "module-inventory", - "reason": "node:readline surface (#3698+) — compile_fail in the 2026-07-03 baseline (tls/zlib/readline compile cluster, possibly runner env); standing per #5917." - }, - "test_gap_disposablestack_2875": { - "issue": "6364", - "added": "2026-07-13", - "category": "bug-open", - "reason": "DisposableStack/Symbol.dispose surface incomplete: `.disposed` returns undefined where Node returns false/true, and the dispose path leaves the adopt/defer callback count at 0. NEWLY VISIBLE, not a regression: DisposableStack is Node 24+, so under CI's old Node 22 pin *node itself* exited non-zero, the harness classified the test `node_fail`, and it was dropped from the gate entirely. Raising the oracle to 26 (.node-version) makes the pre-existing gap observable for the first time. Perry's implementation lives in crates/perry-runtime/src/disposable.rs. Flips to PASS when #6364 lands." + "category": "bug-stale", + "reason": "RE-TRIAGE: tracking issue #3680 is CLOSED but this still fails (audited 2026-08-07, #7582) — needs a new issue. node:v8 Serializer / Deserializer classes still absent from the surface." }, "test_gap_zlib_4917_level": { "issue": "6847", "added": "2026-07-26", - "category": "toolchain", - "reason": "auto-opt pairs the ext-zlib provider archive with a feature-stripped stdlib rebuild: undefined js_zlib_deflate_raw_sync + panic_unwind symbols on a COLD object cache (warm /tmp caches mask it, which is how it went unnoticed). Compiles fine with PERRY_NO_AUTO_OPTIMIZE=1. Not a parity regression." - }, - "test_gap_zlib_fs_assert_2935_2752_2971": { - "issue": "6847", - "added": "2026-07-26", - "category": "toolchain", - "reason": "auto-opt pairs the ext-zlib provider archive with a feature-stripped stdlib rebuild: undefined js_zlib_deflate_raw_sync + panic_unwind symbols on a COLD object cache (warm /tmp caches mask it, which is how it went unnoticed). Compiles fine with PERRY_NO_AUTO_OPTIMIZE=1. Not a parity regression." - }, - "test_gap_3662_node_argvalidation": { - "issue": "6847", - "added": "2026-07-26", - "category": "toolchain", - "reason": "auto-opt pairs the ext-zlib provider archive with a feature-stripped stdlib rebuild: undefined js_zlib_deflate_raw_sync + panic_unwind symbols on a COLD object cache (warm /tmp caches mask it, which is how it went unnoticed). Compiles fine with PERRY_NO_AUTO_OPTIMIZE=1. Not a parity regression." - }, - "test_gap_constants_tail_3683plus": { - "issue": "6847", - "added": "2026-07-26", - "category": "toolchain", - "reason": "auto-opt pairs the ext-zlib provider archive with a feature-stripped stdlib rebuild: undefined js_zlib_deflate_raw_sync + panic_unwind symbols on a COLD object cache (warm /tmp caches mask it, which is how it went unnoticed). Compiles fine with PERRY_NO_AUTO_OPTIMIZE=1. Not a parity regression." - }, - "test_gap_handle_band_object_ops": { - "issue": "6847", - "added": "2026-07-26", - "category": "toolchain", - "reason": "auto-opt pairs the ext-zlib provider archive with a feature-stripped stdlib rebuild: undefined js_zlib_deflate_raw_sync + panic_unwind symbols on a COLD object cache (warm /tmp caches mask it, which is how it went unnoticed). Compiles fine with PERRY_NO_AUTO_OPTIMIZE=1. Not a parity regression." + "category": "bug-stale", + "reason": "RE-TRIAGE, macOS only: #6847 was closed COMPLETED on 2026-07-30 (fixed by #7021, verified on a cold cache) but this test still reproduces its exact signature on macOS/arm64 — `Undefined symbols: _js_zlib_deflate_raw_sync, _js_zlib_inflate_raw_sync` when auto-optimize pairs the ext-zlib provider archive with a feature-stripped stdlib rebuild. Reproduced 3/3 on 2026-08-07 (#7582 audit), so not a flake; its zlib siblings test_gap_zlib_3285_params and test_gap_zlib_fs_assert_2935_2752_2971 both PASS, so it is specific to the raw-sync entry points. Linux passes it (test-parity/gap_snapshot.json), which is why this is scoped rather than global — unscoped it would suppress a passing test on the platform the gate actually runs on. Needs #6847 reopened for macOS, or a new issue.", + "platforms": [ + "macos" + ] } }