Phase 32: general-first re-layering — label axis, markets delegators, panels, contributor registry, provenance ledger (1.15.0)#114
Merged
Conversation
…nclude_* alias train - NEW _contributors.py: module-level registry, @contributor decorator (name, prefix, archive_class, injection_point, key tuple, native_grain), unconditional labels-only firewall (no never_label kwarg), prefix/collision guards, failure-semantics-aware compose_post_join_contributors, features=[...] resolver splitting by injection point. - NEW mostlyright/experimental/__init__.py: public-experimental re-export (pandas api.extensions pattern), D-24 semver-exempt docstring. - core/exceptions.py: ExperimentalFeatureWarning (UserWarning subclass), fires once on first third-party registration; built-ins exempt. - research.py: dataset(features=[...]) param; _merge_features_and_flags folds the four include_* flags into features= with DeprecationWarning (>=2 minors, byte-identical); effective flags drive UNCHANGED built-in wiring; third-party post-join contributors compose after the built-in covariate join. - Migrate 4 built-ins: forecasts (fcst_, pre_aggregation, refetchable), trades (trade_, pre_aggregation, ledger), satellite (sat_, post_join, refetchable), cwop (cwop_, post_join, ledger). features=[]/omitted == today's bytes (parity firewall preserved). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s equivalence, grain broadcast 31 network-free tests covering the binding-amendment contract: - 4 built-ins registered with amended metadata (prefix/archive_class/ injection_point/key/native_grain) - public path at mostlyright.experimental; ExperimentalFeatureWarning fires once on first third-party registration; built-ins exempt - registration guards: bare/non-underscore/label-colliding/builtin-reserved prefix, bad archive_class, bad injection_point, duplicate name - compose aligner: LEFT-join preserves rows/order; uncovered days NaN - UNCONDITIONAL firewall: label/obs_*/cli_* emission + prefixless column rejected loudly (unit + via dataset()) - failure semantics: contribute() raises -> NaN block + attrs['contributor_errors'] + RuntimeWarning, call succeeds; missing-extra (SourceUnavailableError) + firewall violations fail LOUD; empty window -> NaN not error - alias equivalence: features=[] == default bytes; include_satellite=True byte-equal to features=["satellite"]; include_* emits DeprecationWarning - third-party toy contributor: register + compose + firewall rejection + failure-semantics paths via dataset() - grain broadcast: features=[...] at observation grain repeats identically to cli_* repetition - 4 parity-firewall files grep-asserted zero-diff vs phase base Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…BYO aligner
- NEW _labels.py: label-recipe registry ("cli" identity, "daily_extremes"
reusing international.daily_extremes verbatim → WR-05 low-coverage NaN-label
with row PRESERVED + attrs["label_coverage"]), None features-only path.
- BYO-label aligner align_byo_labels(): all six sharp edges — (a) copy-never-
mutate, (b) date-as-column w/ index reset+warning else typed error, (c)
tz-aware settlement-calendar bucketing (02:00Z → prior LST day), (d) station
optional-single/required-panel w/ disagree→error, (e) collision guard vs the
UNION of recipe/feature-prefix/core namespaces, (f) publication_lag requires
published_at (typed error) + per-day as_of = settlement-day market close.
- ALIAS SPLIT (checker-1 blocker): research()/dataset() are now thin wrappers
over shared _dataset_impl; internal _warn_on_default_label sentinel makes
dataset() emit the exact D-22 FutureWarning on omitted label, research() never.
- label composition runs BEFORE the observation-grain broadcast so
granularity="observation" composes with every label mode (cli/None/BYO).
- LabelAlignmentError (typed, reason enum) in core/exceptions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ias-split - test_label_axis.py (24): recipe vocabulary/validation, cli identity, daily_extremes byte-identity to polymarket_settle inputs + WR-05 low-coverage NaN-label/preserved-row, label=None drops cli_* (frame + list paths), omitted==explicit-cli bytes, daily_extremes/None compose with granularity="observation", labels_as_known validation. - test_label_aligner.py (20): all six BYO sharp edges — copy-not-mutate, date-column reset+warning / missing_date error, tz-aware off-by-one bucketing, station optional/disagree/panel-required, collision guard (recipe/core/ feature-prefix), publication_lag requires published_at + withholds late labels. - test_parity.py: pin label="cli" on BOTH parity calls (warning-free live gate); test_dataset_is_research_alias → network-free BYTE-equivalence (shared-body forwarding) after the split; NEW fast-suite pytest.warns assertions — dataset() bare emits exact D-22 FutureWarning once, research() bare + explicit label="cli" never warn. - test_dataset_alias.py / test_dataset_covariates.py: updated the old is-identity asserts to the split contract (distinct wrappers, shared body, same public signature). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…NEW strike parser - markets.kalshi.dataset(ticker, from_date, to_date) is a THIN delegator to core dataset(label="cli", ...) — resolves settlement station via the parity-critical catalog whitelist, zero duplicated join logic. - NET-NEW parse_ticker(): decompose KXHIGHNY-25MAY26-T79 / KXLOWCHI-25MAY26-B40-42 into series/measure/city/strike; series-suffix→catalog-city alias (NY→NYC). - outcome=True appends binary label_outcome (Int64, 0/1/NA) from the parsed strike vs THIS venue's cli_high_f/cli_low_f label column only. - DOCUMENTED settlement comparison: threshold -T<n> resolves YES on value>=strike (INCLUSIVE — boundary equality is YES, matching Kalshi "or above" wording); range -B<lo>-<hi> resolves YES on lo<=value<=hi (both endpoints inclusive). - include_trades defaults False (core trades feature is still a contract-gated stub from 32-01); live trades stay in kalshi_trades.py.
…or + per-date outcome loop - markets.polymarket.dataset(event_id, from_date, to_date) is a THIN delegator to core dataset(label="daily_extremes", ...) — Polymarket resolves on WU/NOAA-WRH extremes (NOT CLI), the venue-correct routing that motivated Phase 32. Station resolved via the existing per-event resolver; zero duplicated join logic. - outcome=True is NETWORK-TOUCHING (documented in docstring): per-settlement-date loop over polymarket_settle (which refetches the Gamma description for expired events to validate RESOLUTION_SOURCE_ALLOWLIST). Event fetched ONCE and cached per call (passed into every polymarket_settle) so a multi-date window makes at most one Gamma fetch. - NET-NEW strike parser: above/below/between <n><C|F> from slug+title, normalized to Celsius (daily_extremes units). DOCUMENTED comparison: above→observed>=strike, below→observed<=strike, between→lo<=observed<=hi (all inclusive). - outcome derived ONLY from this venue's daily_extremes label + its strike; NaN outcome (row preserved) on settle failure, reason stamped in attrs. - new PolymarketStrikeError; dataset + PolymarketStrikeError added to __all__.
…deprecation train - dataset(contract=/contracts=) and include_trades=True now emit a DeprecationWarning pointing at the thin markets layer (markets.kalshi.dataset / markets.polymarket.dataset), which own venue knowledge. Core dataset() stays venue-free. - BEHAVIOR BYTE-IDENTICAL: contract= still raises the existing NotImplementedError; include_trades= still raises its existing ValueError precondition. No column/shape change. Warned ≥2 minors before removal. - Default path (no contract/include_trades) stays warning-free — parity fixtures (label="cli" pinned) remain byte-green and quiet.
…e, deprecation 37 network-free tests (pandas importorskip-guarded): - Delegation call-through: both wrappers forward to a spied core dataset() with the venue-correct label (kalshi→cli, polymarket→daily_extremes) + resolved station — proves zero duplicated join logic via kwargs, not a real join. - Kalshi strike parser: -T/-B fixtures incl. NY→NYC alias, negative strike, malformed/unknown-city/lo>hi errors; >= inclusive boundary equality (79==79→YES); range endpoints both inclusive. - Kalshi outcome: derives from cli_high_f/cli_low_f per series measure; NaN→NA. - Polymarket strike parser: above/below/between with C/F normalization; unparseable raises; above/below/between inclusive-boundary settlement rules. - Polymarket outcome per-date loop: engine value for settlement date, frame fallback elsewhere, settle-failure→NA row-preserved with attrs error stamp. - Core deprecation train: contract=/include_trades= warn pointing at markets layer with behavior byte-identical (NotImplementedError/ValueError preserved); default station+label=cli path stays warning-free.
…on-pool writes - NEW packages/core/src/mostlyright/_provenance.py: append-only JSONL sidecar under $MOSTLYRIGHT_CACHE_DIR/provenance/ recording (source, station, window, fetched_at UTC, row_count, stable order-independent row-window hash) per cache write; per-source sharding; filelock-guarded; size-bounded single-backup rotation at a documented 5 MiB cap; best-effort (never raises on the fetch). - Minimal read API provenance.history(station, window) + lazy mostlyright.provenance accessor for Phase 33 reconcile(). - Wire the ledger into the THREE NARROWED station-pool write paths only (32-04 amendment): weather/cache.py write_cache + write_climate_cache, _fetchers/_iem_mos.py _write_mos_cache (forecast per-runtime). - ZERO read-path behavior change; grep-gate exit 0 (no hosted identifier, no seam).
…ory/grep-gate - 22 tests: record shape + per-source sharding + unsafe-source sanitization, order-independent + type-stable row-window hash, size-bounded single-backup rotation (older records preserved in .1), history() station/window filter across shards + malformed-line skip, best-effort never-raise, multiprocess filelock concurrency (80 lines intact, no torn writes), and a source-level grep-gate assertion that _provenance.py carries no hosted identifier.
…Q15)
- _compose_panel fans out to per-station SINGLE-station _dataset_impl calls and
concatenates: long-format, station column FIRST, caller order preserved,
per-station settlement calendars + per-station label-recipe resolution
(mixed-venue OK; a BYO panel frame is sliced per station by its station column
then the column is dropped so each slice flows through the single-station
aligner). ROW/COLUMN DATA == concat of the per-station calls by construction.
- stations= wired into the mutual-exclusion guard (exactly-one with
station/city/contract/contracts; loud ValueError on any combination, empty
list, duplicate, or non-string entry).
- Panel-only nested attrs["stations"] = {resolved_code: {source, retrieved_at,
label_source, ...}}; single-station attrs stay flat + untouched. Panel is
DataFrame-only (return_type="list" raises).
- stations= added to dataset()/research() wrappers (full explicit signature).
…mixed-venue + attrs - 16 network-free tests (per-station fetch stubs keyed by resolved code so the panel path and the manual single-station loop run identical code): panel row/column data byte-equal to the concat of single-station calls (assert_frame_equal), station column first, caller order preserved, row-count sum; nested attrs["stations"] per resolved station + single-station attrs stay flat; mutual exclusion vs station/city/contract/contracts + empty/dup/ non-string guards; mixed-venue BYO per-station label slice routing + missing station-column error; panel rejects return_type="list".
Mirror the LANDED Python 32-02 label axis in the TS SDK (packages-ts/meta):
- dataset({label}) routed axis: named recipe (cli|daily_extremes|none)
| null (features-only) | BYO row-array through the settlement-calendar aligner.
- cli is IDENTITY (cli_* already on the frame from buildPairs) -> byte-stable
parity by construction; omitted-label default returns cli_* unchanged.
- D-22 bridge + ALIAS SPLIT: research/dataset are now DISTINCT wrappers over one
shared body (_datasetImpl); dataset() fires a ONE-TIME FutureWarning on an
omitted label (fs.ts latch pattern + _resetDefaultLabelWarning test hook),
research() never warns. Explicit label never warns (pinned parity stays clean).
- labels.ts: validateLabel vocabulary gate, applyNoneRecipe (drops cli_*),
applyDailyExtremesRecipe (native daily_extremes_* names, WR-05 low-coverage
nulling), alignByoLabels (the six sharp edges), firewall-inverse collision
guard, LabelAlignmentError (stable reason enum). Label composition runs BEFORE
the observation-grain broadcast (Q8).
- Label vocabulary lands on the ROOT barrel (the only new surface permitted at
root - it touches the default path per the amendment).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parity with the LANDED Python 32-03 strike parsers. Pure, cycle-free venue
knowledge in packages-ts/markets:
- kalshi-ticker.ts: parseTicker decomposes KX{HIGH|LOW}{CITY}-{DATE}-{T<n>|B<lo>-<hi>}
into series/measure/city/strike + resolves the settlement station via the
parity-critical KALSHI_SETTLEMENT_STATIONS whitelist (with the NY->NYC
series-suffix alias on a raw-suffix miss; whitelist untouched). Handles KX/K
prefix, negative strikes, and lo>hi rejection.
- settleKalshiOutcome: INCLUSIVE comparison (DOCUMENTED, mirrors Python 32-03):
threshold value >= strike -> YES; range lo <= value <= hi -> YES. null label
-> null outcome (row preserved). Root barrel (pure, no network).
- polymarket-strike.ts: parsePolymarketStrike (above/below/between <n><C|F> ->
Celsius) + settlePolymarketOutcome (same INCLUSIVE rule for cross-venue
consistency). Lives at the /polymarket subpath (server-side by design).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…paths)
Parity with the LANDED Python 32-01 (registry), 32-04 (panels), 32-03 (markets
sugar). All ship behind LEAN subpaths so the size-budgeted mostlyright root
barrel stays within budget (amendment: only the label axis claims root):
- @mostlyrightmd/meta/experimental (32-01): module-level contributor registry;
registerContributor (the TS analog of the Python @dataset.contributor
decorator, PUBLIC-experimental D-24); UNCONDITIONAL labels-only firewall
(obs_/cli_/wu_/noaa_wrh_ + label/date/station rejected at registration AND on
every emitted column); ONE-TIME ExperimentalFeatureWarning via the fs.ts latch
(+ _resetExperimentalWarning); failure semantics (a throwing contribute()
yields an all-null prefix block + error, the call SUCCEEDS); composePostJoin
seam runs strictly AFTER the built-in join (additive, byte-safe).
- @mostlyrightmd/meta/panels (32-04): datasetPanel({stations}) fans out to
per-station single dataset() calls + concat -> long-format, station FIRST,
caller order preserved, ROW/COLUMN data == the per-station concat BY
CONSTRUCTION; per-station recipes (mixed-venue OK); nested provenance keyed by
resolved code; BYO panel slice routed per station through the single aligner.
- @mostlyrightmd/meta/markets-dataset (32-03): kalshi.dataset (label='cli') /
polymarket.dataset (label='daily_extremes') thin delegators + outcome=true
label_outcome binarizer. Lives in meta (where dataset() is reachable) to avoid
a markets->meta cycle; venue knowledge stays in @mostlyrightmd/markets.
- tsup + exports: three new subpath entries (/experimental /panels
/markets-dataset); root barrel unchanged (0 refs to any of them, verified).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
84 network-free/fetch-mocked tests across markets + meta: - markets/strike-parsers (21): Kalshi parseTicker (threshold/range/negative/NY alias/malformed/unknown-city/lo>hi) + INCLUSIVE boundary-equality settlement; Polymarket above/below/between parse (C/F->C) + inclusive outcome; null->null. - meta/labels (17): validateLabel vocabulary, applyNoneRecipe drop, daily_extremes add + WR-05 low-coverage nulling + row-preserve, alignByoLabels six sharp edges (copy-never-mutate, missing_date, station_disagree, missing_station_panel, collision core/feature-prefix, publication_lag require+withhold). - meta/experimental (10): registration + firewall reject, one-time ExperimentalFeatureWarning latch (+ reset), post-join additive compose, failure semantics (throwing contribute -> all-null block + error, call succeeds), firewall-forbidden emission rejected without shadowing base columns. - meta/label-axis (12): cli identity byte-stability, D-22 dataset() warns ONCE / research() never / explicit never, alias split distinct callables, none/null drop cli_*, BYO LEFT-join, kalshi.dataset delegation + outcome boundary + null. - meta/panels (6): station-first + caller order, concat byte-equality vs per-station loop, nested provenance by resolved code, empty/duplicate guards, mixed-venue recipe resolution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 32-05 label axis (D-22) is the ONLY new surface permitted on the mostlyright
root barrel per the fleet-review amendment (it touches the default path).
Experimental registry, panels, and markets sugar all ship behind lean subpaths
(/experimental /panels /markets-dataset) and add ZERO bytes to the root barrel
(verified: 0 refs to any subpath symbol in dist/index.mjs).
Measured (size-limit, minified + brotlied ESM, all deps):
- meta root barrel: 39.64 KB (pre-32-05) -> 41.61 KB (post) = +1.97 KB,
breaching the 40 KB ceiling by 1.61 KB. All growth is the label axis
(validateLabel/recipes/BYO aligner/LabelAlignmentError wired into dataset()).
- weather: 26.32 KB / 27 KB — UNCHANGED, no bump needed.
- core 8.1/25, markets 3.85/15 — well within budget.
Applying the PRE-AUTHORIZED meta 40->44 KB ceiling (amendment). 41.61 < 44 leaves
~2.4 KB headroom. weather stays at 27 KB (the pre-authorized 27->28 bump is NOT
needed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…protocol, label axis + D-22 bridge, panels, honest with/without) - dataset() reframed as the training-table builder/aligner (NOT a predefined recipe) - honest with/without comparison table (left column always available) - label axis: cli/daily_extremes(native _c names + WR-05 NaN nulling)/None/BYO six rules - D-22 bridge timeline table (1.15 warn -> 2.0 flip; research() never warns) - -W error framing: user-opt-in only (our suites do NOT run warnings-as-errors) - contributor protocol: experimental namespace, unconditional labels-only firewall, NaN+warning failure semantics, refetchable/ledger archive_class enum, grain broadcast - panels (stations= long format), markets thin sugar, provenance ledger
…y firewall = rule #1), label axis + D-22 bridge, D-25 ledger note + PT-3205 - new subsection: contributor registry rules ordered by load-bearing weight; firewall #1 unconditional (obs_/cli_/wu_/noaa_wrh_ prefixes + label/date/station exact); no never_label kwarg - per-leg archive_class {refetchable, ledger} intact inside the registry (satellite/forecasts refetchable; cwop/trades ledger) - failure semantics NaN+warning; fail-loud only for missing-extra/firewall - label axis + D-22 default-label bridge (cli/daily_extremes native _c/None/BYO); research() never warns; parity warning-free via explicit label='cli' pins not filters - new section 11: D-25 fetch-provenance ledger (refetchable legs only) + PT-3205 TS server-side-only note
…nclude_*→features + contract=→markets tables, TS notes - section 1: D-22 FutureWarning explained FIRST + bridge timeline table (1.15 warn → 2.0 flip; research() never warns) + user-opt-in -W error callout (our suites do NOT run warnings-as-errors) - section 2: include_forecast/satellite/cwop/trades → features=[...] table (verified mapping) - section 3: contract=/contracts=/include_trades= → markets.kalshi/polymarket.dataset table (inclusive settlement) - section 5: retrain-event pointer (unchanged rule) - section 6: TS notes — subpath imports (/experimental /panels /markets-dataset), research/dataset alias split, warn latch, ledger N/A server-side (PT-3205)
…ain-table export exists) - corrects the honest-comparison table + see-also: the raw forecast domain table is forecast_nwp() / features=["forecasts"], not a top-level forecasts()
… dataset() = the alignment engine, markets one section - tagline: general-first; dataset() = leakage-safe training-table builder; markets thin convenience layer - What-you-can-build reordered: (1) source-identified domain tables, (2) dataset() the alignment engine (NOT predefined recipe) with label=None worldwide + daily_extremes + BYO, (3) train/infer parity, (4) markets one section (venue-correct cli/daily_extremes routing + outcome, inclusive settlement) - quickstart shows explicit label='cli'; D-22 FutureWarning note - Documentation section links docs/dataset.md, source-identity.md, migration-1.15.md
…alignment
Adjudicated: DEFECT confirmed. docs/dataset.md documents contribute() returning
{(station, settlement_date): {...}} (tuple-keyed), but _align_contribution keyed
only on a plain ISO date string, ignoring spec.key — a third-party contributor
following the docs verbatim had its columns silently dropped. Add
_normalize_contribution_keys to bucket both plain-string and declared-key-tuple
rows to the settlement day. Discriminating test (tuple-keyed toy contributor)
failed pre-fix, passes post-fix; kept as a lock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adjudicated: DEFECT confirmed (Codex correct). On a contribute() exception the
composer set per_day={}, so _align_contribution derived its column universe from
populated days only — an empty universe made the contributor's columns VANISH
rather than appear as the documented all-NaN block. Add a required columns=
declaration to @contributor (third-party required, built-ins derive) and seed the
aligner's column universe from it so a raising/empty contributor still
materializes its all-NaN prefix block. Discriminating test (raising contributor
with declared columns) failed pre-fix, passes post-fix; kept as a lock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l contract Python's Celsius contract wins (the Polymarket settlement engine consumes tmin_c/tmax_c; degC is settlement-correct). TS labels.ts now emits EXACTLY Python's RECIPE_LABEL_COLUMNS['daily_extremes'] set — daily_extremes_tmax_c/tmin_c/tmean_c/n_obs/source_tmax/source_tmin — in Celsius, read from the rollup's pre-conversion degC values (DailyExtreme.tempMinC/MaxC/ MeanC + sourceTmin/Tmax), never an F->C reconversion. Dropped the TS-only precip_in/low_coverage recipe columns; low-coverage mirrors Python (null temp labels, row preserved, n_obs carried). marketsDataset polymarket outcome now compares the degC label directly to the degC strike (removed the (observedF-32)*5/9 reconversion). Tests: exact column-name parity vs the Python list; a degC/degF strike-vs-label unit-confusion fixture; degC value assertions in the weather projection test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ost synonyms Mirror Python _STRIKE_ABOVE_BELOW_RE exactly: over/at least map to above; under/at most map to below; the between connector accepts and|to. A single combined operator regex classifies the synonym (whitespace-normalized) so the vocabulary stays authoritative in one place. Fixture tests per synonym (over/at least -> above, under/at most -> below, 'between .. to ..'). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lement calendar labels.ts alignByoLabels indexed BYO rows by the raw date string, so a tz-aware BYO date never matched the LST-stamped daily row. Mirror Python _bucket_to_settlement_day: a tz-aware BYO date routes through settlementDateFor (a 2025-01-07T02:00Z label buckets to the 2025-01-06 KNYC row); a tz-naive date is an LST day already. Thread tzOverride from research() -> applyLabelToDailyRows -> alignByoLabels. Off-by-one regression test copied from the Python vector (test_label_aligner.py::test_c_tz_aware_off_by_one_lands_in_prior_day_join). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ce per call Python _compose_panel forwarded _warn_on_default_label=False to every per-station call and never warned at the panel level (0 warnings on omitted label); TS datasetPanel fanned out through research() which never warns (also 0). Both now emit the D-22 bridge FutureWarning ONCE per panel call when the label is omitted (never N-per-station), and zero when an explicit label is passed. Python emits at the panel entry then forwards False; TS fires the once-latch at the panel entry then fans out through research(). Tests both SDKs: omitted-label panel warns exactly once; explicit-label panel warns zero. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (no last-write-wins) _labels.py align_byo_labels silently overwrote per_day[settle_day] when two BYO rows bucketed to the same (station, settlement day) — a revised label would drop the original. Now both SDKs raise LabelAlignmentError(reason= 'duplicate_settlement_day') naming the day. Added the reason to the shared lockstep enum (Python VALID_REASONS + TS LABEL_ALIGNMENT_REASONS, D-04). Tests both SDKs: a revision scenario (two rows same day, incl. tz-collapse) raises; distinct days align fine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… stations Adjudicated: DEFECT confirmed. markets.polymarket.dataset resolves an event to its settlement station (which may be an intl ICAO, e.g. EGLC) then delegates to core dataset(label='daily_extremes'). The daily_extremes recipe IS intl-capable (reuses international.daily_extremes), but core _resolve_station is US-only and rejected the intl station before the recipe ran. Add an allow_intl flag to _resolve_station, set ONLY on the daily_extremes recipe path — every other path keeps the strict US-only gate (cli settlement + LST math stay US-only, never weakened). A genuinely-unsupported station still raises loudly. Discriminating test (EGLC intl fixture through the REAL core dataset) failed pre-fix, passes post-fix; kept as a lock. Also updates the parity _halt_after_label harness to accept the new allow_intl kwarg (signature-change blast radius). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er-process (TS) Documentation-only, adjudicated-intended divergence. Python routes the D-22 FutureWarning through stdlib warnings (dedup per call-site per process); TS uses a process-wide console.warn latch. Add one explanatory paragraph to docs/migration-1.15.md so cross-SDK users do not rely on equal warning counts. No behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t object
Seed missing cells in _align_contribution with float("nan") instead of None
so an all-missing declared column materializes as float64 NaN (numeric dtype),
never object. Extends the FIX-4 lock test with float64 dtype assertions on the
failed contributor's block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olumns= Update the era5_ssrd registration example and the decorator signature to include the now-required columns= param (copied from _contributors.py), explain why it is load-bearing for the failure contract, and document the TS ContributorSpec mirror (FIX-B). Verified the example runs via uv run python. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…red columns Mirror Python FIX-4: ContributorSpec gains a required-for-third-party columns declaration (built-ins derive). registerContributor throws a typed ContributorRegistrationError when a third-party omits columns or declares a column outside its prefix namespace. composePostJoinContributors seeds the safe-column universe from the declared set so a contribute() that throws materializes its declared columns as an all-null block, records a contributorErrors entry, and fires a one-time failure warning — the call succeeds. Tests: raising third-party with declared columns yields nulls alongside a working sibling; registration without columns is a typed error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…icing A BYO panelLabels row lacking a station previously forwarded into EVERY per-station slice, leaking the same target to all stations (settlement-grade wrong data): the single-station aligner runs is_panel=false and treats a station-less BYO row as valid, so it never caught the leak. Validate the whole panel frame UP-FRONT in datasetPanel — any station-less row raises LabelAlignmentError(reason='missing_station_panel') before any slicing — mirroring Python _compose_panel's station-required-for-panels rule. slicePanelLabels no longer forwards station-less rows (invariant guaranteed up-front). Tests: panel BYO with one station-less row raises missing_station_panel; single-station BYO without station still composes fine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reorder dataset()/_dataset_impl kwarg validation to run AFTER the
features=[...] <-> include_* effective-flag bridge, not before it.
Previously _validate_research_kwargs ran up-front with the RAW
include_forecast flag, so features=["forecasts"] + forecast_model= was
rejected (the forecasts feature had not yet been merged into the effective
include_forecast). The effective-flag bridge is the single source of truth,
so validation now consumes the merged effective include_forecast:
- dataset(features=["forecasts"], forecast_model="gfs") ==
dataset(include_forecast=True, forecast_model="gfs") byte-for-byte.
- forecast_model WITHOUT the forecasts feature still raises loudly
(effective flag False -> silent-no-op guard fires); error message now
names both entry points (features=["forecasts"] / include_forecast=True).
The relocated call stays BEFORE any network fetch (the merge is pure) and
before backend validation, so the "validate before fetch" contract holds.
The panel dispatch still forwards raw flags to per-station calls, which
merge+validate themselves, so panel warning/validation semantics are
unchanged.
TS parity: N/A. The TS ResearchOptions surface has no features=[...] option
(the 32-01 features<->include_* bridge is Python-only), so there is no
equivalent ordering seam to fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… cli-only) Per the charter, an international catalog station must compose with every CLI-independent label. The US-only station gate exists solely because the cli settlement label is a US product (NWS CLI), so it now binds ONLY to label="cli" instead of "research is US-only". _dataset_impl computes allow_intl = (label != "cli"), so an EGLL-class registry intl station composes with: - label=None (features-only table, no CLI dependency), - label="daily_extremes" (the intl-capable recipe, already allowed), - a BYO label frame (user labels, no CLI dependency). label="cli" stays US-only. _resolve_station's error is reworded to name WHY (CLI is a US product) and to point at the CLI-independent alternatives (label=None / daily_extremes / BYO / international.daily_extremes). For an intl station on a CLI-independent path the downstream IEM CLI fetch 404s and is skipped gracefully (empty climate), so no cli_* columns appear. The PANEL path inherits the rule for free: _compose_panel fans out to per-station _dataset_impl calls that each reach the same gate, so a mixed US+intl panel with label=None composes station-by-station, while a mixed panel with label="cli" fails on the intl station. Parity firewall untouched (merge/, live/_sources.py zero-diff); the US cli path is byte-stable. TS parity: N/A. The TS research surface has no label-scoped intl station gate (isUsStation only selects the GHCNh source; there is no allow_intl equivalent that rejects intl stations), so there is nothing to mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n-out (no wasted fetches)
…slug forms (mirror Python _STRIKE_BETWEEN_RE)
…ser (FIX-alpha/beta, both SDKs)
THE MINUS-SIGN RULE (implemented identically in Python polymarket.py and TS
polymarket-strike.ts): the hyphen is ambiguous — both a slug word-separator
("between-70-...") and a numeric negative sign ("above -5c"). A greedy `-?\d+`
after a `[\s-]*` separator lets the separator swallow the sign, turning
"above -5c" into +5 and FLIPPING every cold-contract settlement.
DECIDED RULE: capture the separator run and the digits SEPARATELY, compute the
sign in code — negative iff sep ends with "--" OR sep ends with (whitespace + "-").
Applied to BOTH the threshold and between regexes.
FIX-beta: remove the TS-only lo-side optional unit group from BETWEEN_RE.
Python has no lo-unit group, so mixed forms like "between 5c and 10f" must
REJECT in TS exactly as Python does (cross-SDK string-set equivalence).
Tests: shared lockstep vector tables (duplicated verbatim in each SDK's test
file with a pointer to its twin) — the five rationale-table cases, the mixed/
lo-unit reject set, plus outcome-level fixtures proving a cold contract settles
CORRECTLY (would have flipped before).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Python-verbatim (FIX-gamma)
The TS TICKER_RE range alternative was `B-?\d+-\d+` (hi with NO optional sign),
which REJECTED `KXLOWMSP-25JAN26-B-10--5` while Python's
`_STRIKE_B_RE` (`^B(?P<lo>-?\d+)-(?P<hi>-?\d+)$`) parses it (lo=-10, hi=-5) — a
cross-SDK divergence that mis-settles negative-range low-temp contracts.
Change the TS range pattern to Python-verbatim `B(-?\d+)-(-?\d+)` and split the
strike token with that exact regex instead of `lastIndexOf('-')` (which would
mis-slice `B-10--5` to (`-10-`, `5`) under the greedy-first-group semantics the
Python regex uses).
Tests: shared lockstep vectors (twin of Python `_KALSHI_RANGE_SIGN_VECTORS`)
covering both-negative / lo-negative-hi-positive / positive-control endpoints,
plus an outcome fixture for a negative-range low-temp contract (both endpoints
inclusive).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rity (FIX-1)
The TS `registerContributor` diverged from Python `_contributors.py`
`_validate_prefix` (:192) + `_register` (:281) on three registration guards,
letting the TS SDK accept registrations Python rejects:
1. prefix must END with "_" — Python `:200` rejects a bare `toy` prefix so
column membership is an unambiguous string-prefix test. TS did not check.
2. duplicate contributor NAME must reject LOUDLY — Python `:293` raises; TS
did `_REGISTRY.set(...)` (silent last-wins overwrite), so a second
registration under the same key silently clobbered the first.
3. third-party registration of a built-in prefix (`fcst_`/`trade_`/`sat_`/
`cwop_`) must reject — Python `:298` guards this; TS had no such check, so
a third-party contributor could shadow a built-in's columns.
Existing empty/reserved-label/columns-required checks are kept. Ordering mirrors
Python: prefix-allowed → duplicate → built-in-prefix → columns-required.
Tests: table-driven accept/reject vectors (REGISTRATION_REJECT_VECTORS /
REGISTRATION_ACCEPT_VECTORS) as the TS twin of the Python registration guards
in test_contributor_registry.py (lockstep twin-pointer comment, same discipline
as the strike vectors), plus a duplicate-name loud-rejection test proving the
original prefix survives (no overwrite).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…FIX-2)
`composePostJoinContributors` treated a firewall-violating emission (a
non-prefixed column, or a forbidden label/obs_*/cli_* column) the SAME as a
runtime contribute() exception — downgrading it to an all-null failure block +
error note. That diverges from Python `_contributors.py`, where the two are
OPPOSITE (`:383` `_assert_columns_clean` RAISES vs `:561` exception → NaN block):
- firewall violation = CORRECTNESS bug = loud → now THROWS the typed
`ContributorFirewallError` immediately (both kinds: non-prefixed column and
forbidden label/obs_*/cli_* column), verbatim-mirroring the two Python
`_assert_columns_clean` messages ("without its required <prefix> prefix" /
"emitted label column ... labels-only firewall, unconditional").
- data/runtime failure = NaN + warning → UNCHANGED: a contribute() that THROWS
still yields the all-null declared block + contributorErrors + one-time
warning, and the call SUCCEEDS (the round-2 FIX-B semantics preserved).
New typed error `ContributorFirewallError` (distinct from the registration-time
`ContributorRegistrationError`) exported from the experimental subpath.
Tests: firewall-violating contributor THROWS for BOTH violation kinds (forbidden
label column + non-prefixed column) — replacing the old "treated as a failure"
test; the runtime-throwing contributor regression (null block + contributorErrors
+ warning, call succeeds) retained as the FIX-B guard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eserved-name strip + enum guards (FIX-1)
Mirror _contributors.py:192-300 registration validation EXACTLY, incl. ordering:
- Reserved label names now reject AFTER stripping the trailing underscore, so
label_/date_/station_ reject (they namespace INTO the label set). Mirrors
Python _validate_prefix `lbl == prefix.rstrip("_")` (:207). FORBIDDEN_EXACT
extended to the full Python _FORBIDDEN_LABEL_EXACT set (obs_high_f, obs_low_f,
obs_high_at, obs_low_at, cli_high_f, cli_low_f) for firewall + strip-stem parity.
- archiveClass in {refetchable, ledger} and injectionPoint in {pre_aggregation,
post_join} are now RUNTIME-validated (closed enums) BEFORE the duplicate-name
and built-in-prefix guards, matching Python's validation order (:283/:288 <
:293/:298). The compile-time unions don't protect a JS caller / as-cast.
- Lockstep vector tables extended in BOTH SDKs (TS REGISTRATION_REJECT_VECTORS +
new ENUM reject/accept vectors + ordering test; Python
test_reserved_label_prefix_after_strip_rejected +
test_enum_validation_precedes_duplicate_name_guard). Python = test additions
only, no source change.
TS: 28 -> 42 tests green, typecheck/biome clean, size unchanged (41.87/44 kB).
Python: 33 -> 35 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ity errors (FIX-2) Runtime-failure SPLIT — mirror Python `compose_post_join_contributors` re-raise of SourceUnavailableError (`_contributors.py:558`). TS composePostJoinContributors now RE-RAISES both SourceUnavailableError AND DataAvailabilityError (the TS availability/config error types at packages-ts/core/src/exceptions/index.ts:175/:233) from contribute(). Python's single availability seam is SourceUnavailableError; the TS availability axis is split across two typed errors, so BOTH propagate loudly. Only an ORDINARY runtime exception falls through to the null-block + contributorErrors + warning path (FIX-B semantics: "data failure = null + warning; config error = loud"). Tests: a contributor throwing SourceUnavailableError → propagates; throwing DataAvailabilityError → propagates; an ordinary Error → null block + contributorErrors + one-time warning (FIX-B regression retained). Python already covers the re-raise (test_missing_extra_fails_loud) — no Python change. TS: 42 -> 45 experimental tests green (full meta suite 276 passed); build, typecheck, biome clean; meta root-barrel size unchanged (41.87/44 kB — the experimental subpath is not in the size-budgeted root barrel). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rship (Python parity) + lockstep noaa_ vectors both SDKs
…identifiers (IEM=code, AWC=icao) — unbreaks label=daily_extremes + polymarket.dataset for US stations
- 3 pyproject.toml + 4 package.json -> 1.15.0 (Python __version__ is metadata-derived; no literal edits) - 4 TS 'export const version' literals 1.14.0 -> 1.15.0 - 7 hello.test.ts version assertions updated in lockstep (root/codegen stay 0.0.0, non-published) - uv.lock refreshed (3 workspace records -> 1.15.0); pnpm-lock.yaml verified no-op (workspace-protocol deps) - inter-package pins untouched (minor bump): Python '>=' floors, TS peerDeps ^0.0.0 rewritten at publish time by release-ts-preflight Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- [1.15.0] dated 2026-07-10: label axis (cli|daily_extremes|None|BYO), thin markets delegators (kalshi/polymarket.dataset + outcome targets + inclusive strike parsers incl. negative-temperature sign rule), station panels, experimental contributor registry (labels-only firewall), features=[...] selector, D-25 fetch-provenance ledger (ships dark) - D-22 bridge documented LOUDLY: bare dataset() FutureWarning, default flips to label=None at 2.0; research() unchanged/byte-stable, never warns; -W error framed as user-opt-in only (suites don't run it; parity pinned via label='cli') - [Unreleased] kept empty; migration doc landed in 32-06 (docs/migration-1.15.md, already final) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
✅ Docs-required check: PASS API-surface change includes docs updates — no reminder needed. API-surface files changed: Docs files changed: |
|
Parity ticket gate: PASSED See |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 32 — general-first re-layering (1.15.0)
dataset()becomes general-first; Kalshi/Polymarket become thin convenience delegators. Ships as 1.15.0 across all 7 dists (3 PyPI + 4 npm).What's in it
research()/dataset()growlabel="cli" | "daily_extremes" | None | <BYO rows>. Default path stays byte-stable (label="cli"implicit); baredataset()withoutlabel=emits a ONE-TIME FutureWarning (bridge; flips at 2.0).label="daily_extremes"= the WU/NOAA-WRH recipe Polymarket settles on; BYO aligner with the six sharp-edge rules (as-of discipline, station guard, collision guard, coverage attrs).markets.kalshi.dataset()(label="cli") andmarkets.polymarket.dataset()(label="daily_extremes") withoutcome=Truecontract targets + inclusive strike parsers, including the negative-temperature sign rule (separator ends--or whitespace+-) — fixes cold-contract settlement flips. Venue-shaped selectors on coredataset()deprecated (≥2 minors).stations=[...]→ stacked panel with nestedattrs["stations"].mostlyright.experimental.contributor— neutral entity/period feature contract, unconditional labels-only firewall (violations throw), runtime-failure → NaN block +attrs["contributor_errors"], availability errors re-raise.features=selector ondataset()._provenance.pyappend-only JSONL on 3 narrowed write paths — the accrual substrate for Phase 33reconcile()./experimental,/panels); PT-3208 records the intended TS lighter-mirror forfeatures=; size budgets green (meta 41.91/44 KB).Review loop
11 Codex (gpt-5.5 high) rounds; 28 P1/P2 findings fixed, 3 disputed findings adjudicated test-first (all real), 1 design dispute recorded (PT-3208); final round independent CLEAN. P3s in
deferred-items.md.Release gates (32-07) — all green
_provenance.pyNOT in seam allowlist)merge/climate.py)__version__= 1.15.0 ×3; D-25 ledger write proof;experimentalimports pandas-freeAfter merge: tags
v1.15.0+vts-1.15.0on the merge SHA → trusted publishing → registry verify ×7.🤖 Generated with Claude Code