Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ jobs:
pnpm vendor:official --measure CMS2FHIRPCSDepScreenAndFollowUp --catalog-id cms2 --strip-elm-annotations $COMPLETE
pnpm vendor:official --measure CMS68FHIRDocumentationCurrentMeds --catalog-id cms68 --strip-elm-annotations $COMPLETE
pnpm vendor:official --measure CMS951FHIRKidneyHealthEval --catalog-id cms951 --strip-elm-annotations $COMPLETE
pnpm vendor:official --measure CMS138FHIRTobaccoScrnCessation --catalog-id cms138 --strip-elm-annotations $COMPLETE

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep uncredentialed fork CI from running CMS138 incomplete

When this job runs for a fork or Dependabot PR, GitHub withholds WORKWELL_VSAC_API_KEY_VENDOR, so lines 170–176 leave $COMPLETE empty. This new invocation then regenerates CMS138 with only the 31 upstream value sets; backend-ts/scripts/vendor-official-measure.mjs:507-515 explicitly says that the resulting artifact cannot run because the missing set requires --complete-terminology. The bundle-only reproducibility check still passes, but the workflow subsequently runs the default pnpm test:official-cases gate at lines 232–234, which now includes CMS138 and therefore reports all 47 cases as errors and exits nonzero. Condition the CMS138 invocation/gate for the uncredentialed path or otherwise provide a credentialless way to retain its validated supplement, so every external PR is not unconditionally red.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct and fixed in 895027d — thank you, this would have turned every external contributor's PR red on merge.

Your trace is exactly right: uncredentialed → $COMPLETE empty → cms138 re-vendored with only upstream's 31 value sets → the supplement has nothing to draw on → 47 errors → nonzero exit. And the bundle-only reproducibility check passes, so nothing upstream of the gate would have caught it.

--allow-missing-terminology skips a measure the CONTEXT cannot resolve, passed by CI only where steps.vendor.outputs.credentialed != 'true' — the same conditional the reproducibility check already splits on, and the same shape official-flip-config.test.ts uses to excuse the capped class.

The opt-OUT is the load-bearing half, so it is the one I pinned hardest: a credentialed run passes nothing and therefore skips nothing, because there an unresolvable value set means a broken artifact — which is the whole point of the gate. There is a test that supplies the SAME unresolvable OID without the flag and asserts the measure still runs, so the accommodation cannot quietly become the default.

Two consequences your comment did not have to reach, both handled:

  1. A skipped run is PARTIAL even though it asked for the full list, so it must not rewrite the committed report — otherwise CI's staleness check would read "current" on the very run that could not produce it. Extended the existing --measure guard rather than adding a second one.
  2. That staleness step is now credentialed-only, since an uncredentialed run deliberately leaves the report alone and the comparison would always differ.

The skip predicate is absentValueSets(artifact, requiredOids(artifact)) — the same one officialRoutingProblems refuses on — so "the gate skipped it" and "routing would refuse it" are one decision, not two that can drift.

Mutation-checked both ways: skipping regardless of the flag, and letting a partial run overwrite the report, each fail exactly one test.

# Re-vendoring must reproduce the COMMITTED artifact byte for byte. Nothing checked this before:
# the manifest's SHA-256 is written by vendor:official about itself, so it could only ever prove
# self-consistency. This proves the artifact in Git is what the pinned upstream actually produces.
Expand Down Expand Up @@ -228,14 +229,30 @@ jobs:
src/wiring/official-flip-config.test.ts \
src/fhir/qrda1-import-official.test.ts \
scripts/valueset-parity.test.mjs
# `--allow-missing-terminology` ONLY where the credential is genuinely absent (fork PRs,
# Dependabot — GitHub withholds the secret there). Without it, the uncredentialed re-vendor above
# produces a cms138 sidecar that omits the value set upstream does not ship, the deck cannot
# resolve it, and every external contributor's PR goes red for a reason unrelated to their change
# (review of #366). The credentialed run on merge passes NOTHING and therefore skips nothing —
# there, an unresolvable value set means a broken artifact, which is what the gate is for.
- name: Run the official MADiE test-case gate
id: gate
run: pnpm test:official-cases
run: |
if [ "${{ steps.vendor.outputs.credentialed }}" = "true" ]; then
pnpm test:official-cases
else
echo "::notice::No VSAC credential here — measures whose upstream bundle omits a value set are SKIPPED rather than failed. The credentialed run on merge covers them."
pnpm test:official-cases --allow-missing-terminology
fi
# The committed report is evidence, so its RESULTS must match what the harness just produced.
# The comparison deliberately ignores the "**Generated:**" line: the harness stamps today's date,
# so a naive `git diff --quiet` would go red the day after every regeneration and train people to
# ignore this job — the worst possible failure mode for a gate.
# Credentialed runs only. An uncredentialed run may skip a measure, which makes it a PARTIAL run —
# the CLI deliberately does not rewrite the committed report there, so comparing against it would
# always differ and would fail every fork PR (review of #366).
- name: Committed evidence report is current (results, not timestamp)
if: steps.vendor.outputs.credentialed == 'true'
working-directory: ${{ github.workspace }}
run: |
report=docs/OFFICIAL_TESTCASE_REPORT_2026-07.md
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/deploy-staging-mieweb.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ jobs:
node scripts/vendor-official-measure.mjs --measure CMS2FHIRPCSDepScreenAndFollowUp --catalog-id cms2 --strip-elm-annotations --complete-terminology
node scripts/vendor-official-measure.mjs --measure CMS68FHIRDocumentationCurrentMeds --catalog-id cms68 --strip-elm-annotations --complete-terminology
node scripts/vendor-official-measure.mjs --measure CMS951FHIRKidneyHealthEval --catalog-id cms951 --strip-elm-annotations --complete-terminology
node scripts/vendor-official-measure.mjs --measure CMS138FHIRTobaccoScrnCessation --catalog-id cms138 --strip-elm-annotations --complete-terminology
- name: The committed artifact is reproducible from its pin
run: git diff --exit-code backend-ts/measures/official
- uses: docker/login-action@v3
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/deploy-twh-mieweb.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ jobs:
node scripts/vendor-official-measure.mjs --measure CMS2FHIRPCSDepScreenAndFollowUp --catalog-id cms2 --strip-elm-annotations --complete-terminology
node scripts/vendor-official-measure.mjs --measure CMS68FHIRDocumentationCurrentMeds --catalog-id cms68 --strip-elm-annotations --complete-terminology
node scripts/vendor-official-measure.mjs --measure CMS951FHIRKidneyHealthEval --catalog-id cms951 --strip-elm-annotations --complete-terminology
node scripts/vendor-official-measure.mjs --measure CMS138FHIRTobaccoScrnCessation --catalog-id cms138 --strip-elm-annotations --complete-terminology
# The vendored artifact must not have changed — only the gitignored sidecar is new. A non-empty
# diff means the committed bundle does not match its pin, which must never reach a deploy image.
- name: The committed artifact is reproducible from its pin
Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,7 +368,8 @@ here, so the nightly ALL_PROGRAMS run exercises the flip unprompted. Rollback =
(ADR-040 makes `eval_state` invalidate by construction).

**M-A WAVE 2 (ADR-047): CMS2, CMS68 and CMS951 are vendored, MADiE-gated and ROUTABLE — none is routed.**
The gate is now **231/231** across five measures (55+66+36+19+55, 0 unexpected, 0 errors) and drives the
**CMS138 joined them 2026-07-31 (ADR-053) — the gate is now 278/278 across SIX measures.**
The gate was **231/231** across five measures (55+66+36+19+55, 0 unexpected, 0 errors) and drives the
harness, the sparse checkout and the committed-report predicate off `OFFICIAL_GATED_MEASURES` instead of a
hardcoded pair — all three silently stopped meaning "the full gate" the moment a third measure existed.
**Three of the six did NOT onboard, each for a different reason:** CMS138 scores **0/47 with 47 errors**
Expand Down
1 change: 1 addition & 0 deletions backend-ts/measures/official/cms138/bundle.json

Large diffs are not rendered by default.

62 changes: 62 additions & 0 deletions backend-ts/measures/official/cms138/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"catalogId": "cms138",
"measureName": "CMS138FHIRTobaccoScrnCessation",
"version": "1.0.000",
"cmsId": "138FHIR",
"url": "https://madie.cms.gov/Measure/CMS138FHIRTobaccoScrnCessation",
"status": "active",
"effectivePeriod": {
"start": "2026-01-01",
"end": "2026-12-31"
},
"scoring": "proportion",
"populationBasis": "boolean",
"improvementNotation": "increase",
"populations": [
"initial-population",
"denominator",
"denominator-exclusion",
"numerator"
],
"source": {
"repo": "cqframework/dqm-content-qicore-2025",
"ref": "ca4b49516de4cbed9f92bfb7c35d97b1bf1022ab",
"path": "bundles/measure/CMS138FHIRTobaccoScrnCessation/CMS138FHIRTobaccoScrnCessation-bundle.json",
"rawSha256": "sha256:1056b796f19ee79cb76a6c76022f511efcc0e7b6e2fa18c90637ae9669ab9989"
},
"reduction": {
"keptResourceTypes": [
"Measure",
"Library"
],
"libraryContentTypes": [
"application/elm+json"
],
"strippedNarratives": true,
"strippedElmAnnotations": true,
"strippedValueSets": true,
"rawBytes": 15842294,
"vendoredBytes": 2343564
},
"terminology": {
"file": "terminology.json",
"valueSets": 32,
"codes": 549,
"truncated": [],
"completion": {
"source": "vsac",
"manifest": "http://cts.nlm.nih.gov/fhir/Library/ecqm-fhir-update-2025",
"valueSets": [
{
"oid": "2.16.840.1.113883.3.526.3.1278",
"reason": "absent-upstream",
"had": 0,
"now": 4,
"declaredTotal": null
}
]
},
"sha256": "sha256:e2f8b334c9b352f419eeaec736ce9b914b384bf7a34f1453d50db458aea417d8"
},
"sha256": "sha256:219b52a38363ac6968d47b0d51b6b45656da6b01a9a9d2496bd05aab25280eb4"
}
102 changes: 102 additions & 0 deletions backend-ts/src/run/cli/official-cases.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ test("main loads a selected measure without overwriting the committed combined r
const fakeLoaded = { measure: "cms125" };
const fakeRun = {
measure: "cms125",
// Required on OfficialMeasureRun since ADR-053's supplement. The double is cast `as never`, so the
// type does not enforce it here — and omitting it made `main` throw and return exit 2, which read
// as "the CLI refused this configuration" rather than "the fixture is incomplete".
supplementedOids: [],
summary: {
total: 66,
expectedAgreements: 66,
Expand Down Expand Up @@ -70,6 +74,7 @@ test("main loads a selected measure without overwriting the committed combined r
// "stub the world" would parse a 2.4MB bundle and expand 26 value sets, taking a different path
// on a machine that has the fetched sidecar than on one that does not.
runtimeTerminology: () => Promise.resolve({ reason: "stubbed" }),
unresolvableOids: () => [],
runDraftDrift: () => Promise.resolve({ total: 66, changedCases: 0, errors: 0 } as never),
writeReport: (path: string, markdown: string) => {
writes.push({ path, markdown });
Expand All @@ -90,6 +95,7 @@ test("main runs the CMS122 vendored-draft drift stretch after the official batch
const fakeLoaded = { measure: "cms122" } as never;
const fakeRun = {
measure: "cms122",
supplementedOids: [],
summary: { total: 55, expectedAgreements: 55, referenceAgreements: 0, unexpectedMismatches: 0, errors: 0 },
};
// changedCases MUST be 0 now: since PR-5 both sides of this comparison are v1.0.000, so any change
Expand All @@ -103,6 +109,7 @@ test("main runs the CMS122 vendored-draft drift stretch after the official batch
load: () => fakeLoaded,
run: () => Promise.resolve(fakeRun as never),
runtimeTerminology: () => Promise.resolve({ reason: "stubbed" }),
unresolvableOids: () => [],
loadDraftBundle: (path) => {
assert.equal(path, resolve(testCwd, "measures", "official", "cms122", "bundle.json"));
return { resourceType: "Bundle", entry: [] } as never;
Expand Down Expand Up @@ -184,3 +191,98 @@ test("readArtifactIdentity hashes the bytes on disk, not the manifest's claim ab
// A missing artifact is absent identity, not a thrown gate.
assert.equal(readArtifactIdentity(resolve(malformed, "nope.json")), undefined);
});

/**
* ADR-053 / review of #366. GitHub withholds `WORKWELL_VSAC_API_KEY_VENDOR` from fork and Dependabot
* PRs, so CI re-vendors without `--complete-terminology`. For a measure whose upstream bundle omits a
* value set, the regenerated sidecar omits it too — the deck cannot resolve it and reports every case
* as an error. Unhandled, that turns every external contributor's PR red for a reason unrelated to
* their change.
*
* The accommodation is opt-in, and the opt-OUT is the load-bearing half: a credentialed run must never
* skip, because there an unresolvable value set means a broken artifact — which is what the gate is for.
*/
test("--allow-missing-terminology SKIPS a measure this context cannot resolve, and says so", async () => {
const module = await import("./official-cases.ts");
const ran: string[] = [];
const errors: string[] = [];
const writes: unknown[] = [];
const code = await module.main(["--content-dir", "fixtures", "--allow-missing-terminology"], {
cwd: resolve("test-repo", "backend-ts"),
load: (_dir: string, measure: string) => ({ measure }) as never,
run: (loaded: { measure: string }) => {
ran.push(loaded.measure);
// The REQUIRED deck size for this measure, not 1: `exitCodeForRuns` fails a run whose deck
// shrank, so a fixture with total=1 exits 1 for a reason that has nothing to do with the skip.
const total = module.REQUIRED_OFFICIAL_CASE_COUNTS[loaded.measure] ?? 1;
return Promise.resolve({
measure: loaded.measure,
supplementedOids: [],
summary: { total, expectedAgreements: total, referenceAgreements: 0, unexpectedMismatches: 0, errors: 0 },
} as never);
},
render: () => "report",
sourceRevision: () => "rev",
loadDraftBundle: () => ({}) as never,
artifactIdentity: () => undefined,
runDraftDrift: () => Promise.resolve({ total: 1, changedCases: 0, errors: 0 } as never),
runtimeTerminology: () => Promise.resolve({ reason: "stubbed" }),
unresolvableOids: (measure: string) => (measure === "cms138" ? ["2.16.840.1.113883.3.526.3.1278"] : []),
generatedDate: "2026-07-31",
writeReport: (path: string, markdown: string) => writes.push({ path, markdown }),
log: () => undefined,
error: (m: string) => errors.push(m),
});

assert.equal(code, 0, "a skipped measure must not fail the run in this context");
assert.ok(!ran.includes("cms138"), "cms138 must not be executed at all");
assert.ok(ran.length > 0, "every other measure must still run — this is a skip, not a bypass");
assert.match(errors.join(" | "), /CMS138 SKIPPED/);
assert.match(errors.join(" | "), /credentialed run does NOT skip/);
// A run missing a measure is PARTIAL, so it must not overwrite the committed evidence — otherwise
// CI's staleness check reads "current" on the very run that could not produce it.
assert.deepEqual(writes, [], "a partial run must not rewrite the committed report");
assert.match(errors.join(" | "), /PARTIAL RUN/);
});

test("WITHOUT the flag the same measure still runs — the gate is not quietly weakened", async () => {
const module = await import("./official-cases.ts");
const ran: string[] = [];
const code = await module.main(["--content-dir", "fixtures"], {
cwd: resolve("test-repo", "backend-ts"),
load: (_dir: string, measure: string) => ({ measure }) as never,
run: (loaded: { measure: string }) => {
ran.push(loaded.measure);
// The REQUIRED deck size for this measure, not 1: `exitCodeForRuns` fails a run whose deck
// shrank, so a fixture with total=1 exits 1 for a reason that has nothing to do with the skip.
const total = module.REQUIRED_OFFICIAL_CASE_COUNTS[loaded.measure] ?? 1;
return Promise.resolve({
measure: loaded.measure,
supplementedOids: [],
summary: { total, expectedAgreements: total, referenceAgreements: 0, unexpectedMismatches: 0, errors: 0 },
} as never);
},
render: () => "report",
sourceRevision: () => "rev",
loadDraftBundle: () => ({}) as never,
artifactIdentity: () => undefined,
runDraftDrift: () => Promise.resolve({ total: 1, changedCases: 0, errors: 0 } as never),
runtimeTerminology: () => Promise.resolve({ reason: "stubbed" }),
// Reports the SAME unresolvable OID. Without the flag it must be ignored and the measure run, so a
// credentialed CI job sees the real 47 errors rather than a green skip.
unresolvableOids: (measure: string) => (measure === "cms138" ? ["2.16.840.1.113883.3.526.3.1278"] : []),
generatedDate: "2026-07-31",
writeReport: () => undefined,
log: () => undefined,
error: () => undefined,
});

assert.equal(code, 0);
assert.ok(ran.includes("cms138"), "without the flag the measure MUST be executed");
});

test("parseArgs accepts --allow-missing-terminology and defaults it off", async () => {
const module = await import("./official-cases.ts");
assert.equal(module.parseArgs([]).allowMissingTerminology, undefined, "off unless asked for");
assert.equal(module.parseArgs(["--allow-missing-terminology"]).allowMissingTerminology, true);
});
Loading
Loading