Fail closed on exact-treatment scoping and seed the baseline in the arm's own frame - #36
Open
github-actions[bot] wants to merge 3 commits into
Open
Fail closed on exact-treatment scoping and seed the baseline in the arm's own frame#36github-actions[bot] wants to merge 3 commits into
github-actions[bot] wants to merge 3 commits into
Conversation
added 3 commits
July 31, 2026 13:54
…D coalescing, and an engine that is not inert
Rebased onto main so one branch carries both rounds (the earlier fixes3
branch did not merge; its content is included here unchanged).
EXPOSURE TO A PRICE MEANS BEING SHOWN THAT PRICE. qualified_exposure counted
free preflight_run events toward the PAID deep-preflight experiment, so the
engine could reach its denominator on callers who were never offered
anything and then halve a price nobody saw. There is now an explicit
impression boundary: a canonical `paid_offer_shown` event carrying
`challenged_operation`, recorded identically on HTTP, MCP and A2A. Two
impressions qualify, distinguished by `impression` so they are never
conflated: a 402 quote, and a price DISPLAYED in a successful response where
there is no 402 (a watch is provisioned free and priced per cycle, so its
response is the only moment its price is shown - previously a watch pricing
experiment could never become decidable at all). Adjacent free products never
count, and a free-tier call of a paid operation's shape is not an impression.
THE IMPRESSION WAS DEFINED CORRECTLY AND POPULATED ON NO REAL PATH. Events
moved; distinct ACTORS - the number the engine gates on - stayed at zero.
* HTTP recorded actor=None; the middleware now binds the same stable,
privacy-safe actor used for demand dedupe into a request-scoped
contextvar the challenge path reads.
* Unauthenticated MCP callers collapsed into the literal actor "mcp". The
actor is now a purpose-scoped hash of advertised clientInfo. We do NOT
invent identities: a caller advertising nothing distinguishing is
"mcp:unidentified" with actor_distinct=False and is excluded from actor
thresholds rather than given a fake unique id per call.
ONE DID IS ONE SUBJECT. The module documented endpoint-then-identity dedupe
and performed endpoint dedupe only, inflating the one number the index is
judged on. Endpoints declaring the same did:key are now coalesced at ingest
into a canonical entry with `alias_endpoints` retaining their own provenance
and last observation; summaries report subjects and endpoints separately.
reconcile_identities() migrates existing duplicates - oldest wins,
deterministic, idempotent, no-op when there is nothing to merge. OPERATOR
equivalence is never inferred from names, domains or contact strings:
guessing would launder one party's evidence into another's, a worse error
than the one being fixed.
THE ENGINE HAD NO EXPERIMENT. GET /commercial on deployed 061dcea returned
experiments: {} - nothing ever created one, so the engine evaluated an empty
dict forever. seed_defaults() runs on the real cycle path and is idempotent
three ways: never overwrites an existing experiment (a restart cannot reset
its window or baseline), never seeds one whose price an operator has pinned,
and seeds exactly ONE. It does not fabricate exposure - a seeded experiment
nobody has been offered reports insufficient_evidence, which is the honest
state.
Also: remote ingest is bounded default-ON for CLEARED_SOURCES only (the
documented public MCP Registry API) with GUILD_INDEX_INGEST=0 retained as a
no-deploy kill switch; excluded sources are named with their exact gate
rather than silently omitted.
Tests: 1150 passed, 9 skipped. The transport suite drives real FastAPI,
MCP-tool and A2A JSON-RPC paths rather than hand-built events.
The 061dcea auto-revert was a DEPLOY-TIMEOUT FALSE NEGATIVE: the gate stopped
waiting before Render finished, production then served the merged SHA and was
healthy. The revert branch is inert (the ship job skips ship/revert-*) and
must not be merged.
Experimental-validity defect, and the most consequential of the round: after
repricing, evaluate() reset started_at and the baseline, but
qualified_exposure() still counted ALL historical impressions for the
operation. So ten callers seeing 20 credits and not buying triggered a cut to
10 - and on the next cycle those same ten old-price impressions were counted
again to justify cutting 10 to 5, though nobody had been shown 10. Left
alone, the loop repeatedly optimises on stale treatment data and walks a
price to zero without ever testing any of the prices in between.
An arm is (window, price), and the three now move together:
* `tested_price_credits` is persisted on define/seed and updated on every
reprice, alongside started_at and a re-taken baseline;
* exposure used by evaluate requires event.at >= started_at AND
event.price_credits == tested_price_credits - an impression is evidence
about the price that was actually displayed, and about nothing else;
* completions carry the credits actually QUOTED, not today's price. HTTP
and MCP stamp the price at request time; A2A uses the credits_cost stored
on the payment task, so a payment quoted under the old arm and settled
late cannot promote the new one. The money is real - it is simply
evidence about the price the payer was shown.
Event-time parsing FAILS CLOSED: an event whose timestamp cannot be parsed is
excluded, never admitted. An unreadable date must not be able to decide a
price.
Scoping applies to EXPERIMENT EVALUATION ONLY. The portfolio view at
/commercial is deliberately unscoped and still reports everything that
happened, with a test proving it was not narrowed by this change.
Tests: 1157 passed, 9 skipped (7 new), including the exact regression - old
price impressions cannot decide the new window, new-price impressions can,
pre-window and unparseable events are excluded, and an old-quoted payment
cannot promote the new price.
…rm's own frame Two edge cases in 2d86cd7, both of which failed OPEN. 1. EXACT-TREATMENT SCOPING ADMITTED UNPRICED COMPLETIONS. The filter excluded a completion only when price_credits was present AND different, so a paid completion with no recorded price passed straight through and could promote a price it was never quoted at. When a tested price is supplied, price_credits must now EXIST and equal it. Excluding a real sale understates us; admitting an unattributable one is the failure this whole round exists to prevent. 2. THE SEED BASELINE WAS IN A DIFFERENT FRAME FROM THE COMPARISON. seed_defaults took an ALL-TIME baseline while evaluate() measures the arm (since started_at, at the tested price). Any historical revenue therefore sat in the baseline while a genuine new-arm sale sat in the metrics, and the sale read as "no movement" - an experiment that could never promote on a real result. The baseline is now taken in the arm's own window and at its own price, so an arm opens at zero, which is the truth about a window that has just started. Tests: 1159 passed, 9 skipped (2 new) - a missing-price completion is excluded, and historical revenue before seeding cannot suppress promotion from a genuine new-arm sale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Machine-operated ship branch
ship/index-final3.The ship workflow dispatches the full
cimatrix, waits for it, merges only the certified combined state with main, then runs the deployment-aware release gate against the merged SHA — with automatic certified revert on a red gate. See .github/workflows/ship.yml.