test(compat): pinned-Node builtin-module compatibility matrix #7074
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| name: Node Compat Matrix Guard | ||
|
|
||
| # Gates scripts/node_compat_matrix.mjs --check: a breadth sweep over every | ||
| # require("module").builtinModules entry, both import forms (M and node:M), | ||
| # against a PINNED, SRI-verified Node oracle (external-tools.json tools.node, | ||
| # currently 26.5.1). It compares Perry's export-SHAPE fingerprint per module | ||
| # to the oracle's and FAILS if any baselined cell got strictly worse or a | ||
| # prefix-parity invariant broke (test-parity/node-compat-matrix.baseline.json). | ||
| # The systematic form of tracker #812. Behavioral parity stays in the | ||
| # node-suite (node-suite-guard.yml) — this is shape breadth, one pinned oracle. | ||
| # | ||
| # Its OWN job on purpose: the runner downloads the pinned Node dist tarball | ||
| # (~40MB) and verifies it, which must not slow the main cargo-test job. The | ||
| # node used by setup-node here only EXECUTES the .mjs; the oracle Node is the | ||
| # pinned dist the runner fetches + SRI-verifies itself. | ||
| # | ||
| # Decoupled from the (not-yet-enabled) merge queue like node-suite-guard.yml: | ||
| # runs nightly + on demand today, and the merge_group trigger is INERT until a | ||
| # maintainer enables the merge queue in branch protection. | ||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| # Nightly, offset from node-suite-guard (37 4) and the Core Subset Radar | ||
| # (17 3) to avoid overlap. | ||
| - cron: "57 4 * * *" | ||
| merge_group: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: node-compat-matrix-${{ github.ref }} | ||
| cancel-in-progress: false | ||
|
|
||
| env: | ||
| RUSTC_WRAPPER: sccache | ||
| SCCACHE_GHA_ENABLED: "true" | ||
| SCCACHE_CACHE_SIZE: "2G" | ||
| CARGO_INCREMENTAL: "0" | ||
|
|
||
| jobs: | ||
| node-compat-matrix: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 120 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| with: | ||
| # Read-only job (build + verify); don't leave the GITHUB_TOKEN in | ||
| # the local git config (least privilege — OWASP / CodeRabbit). | ||
| persist-credentials: false | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
|
|
||
| - name: Start sccache | ||
| uses: mozilla-actions/sccache-action@v0.0.10 | ||
|
|
||
| - uses: Swatinem/rust-cache@v2 | ||
| with: | ||
| shared-key: "${{ runner.os }}-perry" | ||
| save-if: ${{ github.ref == 'refs/heads/main' }} | ||
|
|
||
| - name: Setup Node.js (executes the runner; oracle is the pinned dist) | ||
| uses: actions/setup-node@v7 | ||
| with: | ||
| # Single source of truth: .node-version at the repo root. This node | ||
| # only RUNS node_compat_matrix.mjs; the compat oracle is the pinned | ||
| # Node the runner downloads + SRI-verifies from external-tools.json. | ||
| node-version-file: .node-version | ||
|
|
||
| - name: Build Perry release binary | ||
| run: cargo build --release -p perry -p perry-runtime -p perry-stdlib -p perry-runtime-static -p perry-stdlib-static | ||
|
|
||
| - name: Node builtin compat matrix guard (--check, pinned oracle) | ||
| # Fails (exit 1) if any baselined cell got strictly worse or a | ||
| # prefix-parity invariant broke; improvements are accepted. The runner | ||
| # downloads + SRI-verifies the pinned Node dist under .cache/node-pin/. | ||
| run: | | ||
| set -euo pipefail | ||
| echo '### Node builtin compat matrix guard' >> "$GITHUB_STEP_SUMMARY" | ||
| echo '```' >> "$GITHUB_STEP_SUMMARY" | ||
| node scripts/node_compat_matrix.mjs --check \ | ||
| | tee -a "$GITHUB_STEP_SUMMARY" | ||
| echo '```' >> "$GITHUB_STEP_SUMMARY" | ||
|
Comment on lines
+74
to
+84
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win Assert that the full sweep actually ran.
As per coding guidelines, the CI gate must assert that its subject actually ran. 🤖 Prompt for AI AgentsSource: Coding guidelines |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| **Node builtin-module compatibility matrix** (`scripts/node_compat_matrix.mjs`) — a breadth harness that sweeps every `require("module").builtinModules` entry, both import forms (`M` and `node:M`), against a pinned, SRI-verified Node oracle and compares Perry's export-SHAPE fingerprint (sorted `name:typeof` over the module namespace + the default export's typeof) to Node's. This is the systematic version of tracker #812; deep behavioral parity stays in the hand-authored node-suite (`run_parity_tests.sh`). | ||
|
|
||
| - **Pinned oracle**: Node **26.5.1** (latest CURRENT stable) added to `external-tools.json` under `tools.node` with a per-platform `sha512` SRI for darwin/linux/win (arm64+x64), each cross-checked against nodejs.org `SHASUMS256.txt` at pin time. The runner downloads + verifies + caches it under `.cache/node-pin/` (gitignored); `scripts/soak/external-tools.mts` learns a `node-dist` release type so its pin gate stays green without treating nodejs.org as a github release host. | ||
| - **Fast loop for LLMs/humans**: `node scripts/node_compat_matrix.mjs --module fs` (both forms, one module); `--module a,b,c`, `--method`, and `--only mod.export` narrow to a sub-second inner loop. `--check` gates against `test-parity/node-compat-matrix.baseline.json` (own CI job, `.github/workflows/node-compat-matrix.yml`), `--update-baseline` refreshes it; a `--module` selector scopes both to that slice. | ||
| - **Prefixed/unprefixed invariant** verified per module (Perry's `is_native_module` strips `node:`, so `M` and `node:M` must agree). | ||
| - **Baseline on darwin-arm64**: 58 builtins probed — 21 both-forms exact-shape match, 35 shape-diff (partial surfaces; recorded, not regressions), 0 perry-unresolved, **0 prefix-parity divergences**. Findings recorded (not fixed here, per infra-PR scope): `sea`/`sqlite`/`test`/`test/reporters` are `node:`-prefix-only in Node 26 (the bare specifier throws) yet Perry resolves the unprefixed form too; `trace_events` matches Node fully but is absent from `perry-api-manifest::{NATIVE_MODULES,NODE_SUBMODULES}` (works-but-unclaimed). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| # Node builtin-module Compatibility Matrix | ||
|
|
||
| Perry reimplements the `node:*` module surface natively. The **compatibility | ||
| matrix** (`scripts/node_compat_matrix.mjs`) measures — against a *pinned, | ||
| verified* Node — how faithfully Perry reproduces each builtin's **export | ||
| shape**, for **both** import forms (`M` and `node:M`). | ||
|
|
||
| Its value is **breadth**: every builtin, both forms, one pinned oracle. It is | ||
| the systematic version of tracker | ||
| [#812](https://github.com/skelpo/perry/issues/812) ("42-module behavioral | ||
| matrix"). Deep *behavioral* parity lives in the hand-authored node-suite | ||
| (`run_parity_tests.sh`); this harness is a wide, shallow **shape** sweep. | ||
|
|
||
| ## To check one module fast | ||
|
|
||
| ```bash | ||
| node scripts/node_compat_matrix.mjs --module fs | ||
| ``` | ||
|
|
||
| That is the command to reach for while iterating on a single builtin. The | ||
| pinned Node download happens once and is cached under `.cache/node-pin/`, so | ||
| subsequent runs are just Perry compile + run. Narrow further: | ||
|
|
||
| ```bash | ||
| node scripts/node_compat_matrix.mjs --module fs,path,crypto # a few modules | ||
| node scripts/node_compat_matrix.mjs --module fs --method readFileSync,promises # only these exports | ||
| node scripts/node_compat_matrix.mjs --only fs.readFileSync,path.join # combined mod.export form | ||
| ``` | ||
|
|
||
| A `--method`/`--only` subset narrows the fingerprint to those exports for a | ||
| sub-second "did my change fix `node:fs.readFileSync`?" loop. Because it changes | ||
| the fingerprint semantics, it is a **print-only diagnostic** — it is refused | ||
| for `--check`/`--update-baseline`. | ||
|
|
||
| ## Full sweep and the CI gate | ||
|
|
||
| ```bash | ||
| node scripts/node_compat_matrix.mjs # whole matrix + summary table | ||
| node scripts/node_compat_matrix.mjs --check # exit 1 on regressions vs the baseline | ||
| node scripts/node_compat_matrix.mjs --update-baseline # rewrite the committed baseline | ||
| ``` | ||
|
|
||
| The harness needs the release binary (`cargo build --release -p perry`). The | ||
| baseline lives at `test-parity/node-compat-matrix.baseline.json`; the CI job | ||
| `.github/workflows/node-compat-matrix.yml` runs `--check` nightly in its own | ||
| job (so the pinned-Node download never slows the main test job). | ||
|
|
||
| `--check` fails when a baselined cell got **strictly worse** (per a severity | ||
| order: `match` → `shape-diff` → `perry-unresolved`) or when a **prefix-parity** | ||
| invariant that previously held broke. Improvements are always accepted and are | ||
| folded in by `--update-baseline`. A `--module` selector scopes | ||
| `--check`/`--update-baseline` to just that slice — a single-module refresh | ||
| never rewrites the whole baseline. | ||
|
|
||
| ## The pin | ||
|
|
||
| The oracle is the **official nodejs.org dist tarball** for the host platform, a | ||
| *binary* pin (not a source checkout — we measure shape against the runtime Node | ||
| actually ships). It is recorded in `external-tools.json` under `tools.node` | ||
| with a `sha512` SRI per platform, matching that file's existing convention. The | ||
| runner downloads it, verifies the SRI, and caches it under `.cache/node-pin/` | ||
| (gitignored). nodejs.org also publishes `SHASUMS256.txt` (sha256), which is | ||
| cross-checked against every pinned asset at pin time. | ||
|
|
||
| Currently pinned: **Node 26.5.1** (the latest CURRENT stable). This is a | ||
| separate concern from `.node-version` (26.5.0), which pins the gap-suite / | ||
| node-suite oracle; the compat matrix carries its own "latest stable" pin. | ||
|
|
||
| ## The fingerprint | ||
|
|
||
| For a module `M` and a form, the probe does: | ||
|
|
||
| ```ts | ||
| import * as ns from "M" // or "node:M" | ||
| // sorted list of `<exportName>:<typeof export[name]>` over Object.keys(ns), | ||
| // plus `default:<typeof ns.default>`, wrapped in a __FP__...__FP__ sentinel. | ||
| ``` | ||
|
|
||
| Two fingerprints are **equal** iff the two module namespaces have the same | ||
| export names with the same `typeof` for each, and the same default-export | ||
| `typeof`. It is a **shape** fingerprint (names + typeofs), not deep behavior. | ||
| The sentinel line means environmental warnings on stdout/stderr never touch the | ||
| compare — no output-normalization needed. | ||
|
|
||
| Each `(module, form)` cell gets a status: | ||
|
|
||
| | status | meaning | | ||
| | --- | --- | | ||
| | `match` | Perry fingerprint == oracle fingerprint | | ||
| | `shape-diff` | both resolved, fingerprints differ (a real shape gap) | | ||
| | `perry-unresolved` | Node resolved, Perry did not compile/run | | ||
| | `perry-extra` | Perry resolved a form the oracle did not | | ||
| | `both-unresolved` | neither resolved (neutral) | | ||
| | `skip` | curated skip — see `test-parity/node-compat-matrix.skip.json` | | ||
|
|
||
| Modules that cannot be meaningfully fingerprinted by a bare `import * as m` | ||
| (side-effectful on import, or shape depends on constructor args) go in the skip | ||
| JSON **with a reason** rather than silently passing. | ||
|
|
||
| ## The prefixed / unprefixed invariant | ||
|
|
||
| Node treats `M` and `node:M` identically for builtins, and Perry's | ||
| `is_native_module` strips the `node:` prefix — so the two forms **must** agree. | ||
| The runner computes both and records `prefixParity`. A `false` there is a **real | ||
| Perry bug** and fails `--check` if it previously held. | ||
|
|
||
| ## Bumping the pinned Node | ||
|
|
||
| 1. Edit `tools.node.version` in `external-tools.json` and refresh the | ||
| per-platform `sha512` SRI (download each dist tarball, verify its sha256 | ||
| against that version's `SHASUMS256.txt`, then record the recomputed sha512). | ||
| 2. `node scripts/node_compat_matrix.mjs --update-baseline`. | ||
| 3. **Review the diff.** A Node bump legitimately changes fingerprints (new | ||
| exports, typeof changes); confirm the deltas are Node's, not Perry | ||
| regressions, before committing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Do not compare the Darwin baseline from an Ubuntu runner.
The committed baseline is
darwin-arm64, but this job runs onubuntu-latest. The suppliedcheckAgainstBaselineimplementation compares module cells without rejecting a platform mismatch, so this can produce false regressions or miss platform-specific regressions. Use a matching Darwin arm64 runner or make the baseline/check platform-specific.Based on the supplied PR objectives and harness baseline contract.
🤖 Prompt for AI Agents