From 19c96ccc913498b220f07dece879be5b0d9dd099 Mon Sep 17 00:00:00 2001 From: helloiamvu Date: Wed, 15 Jul 2026 10:06:34 +0200 Subject: [PATCH 1/4] docs: headline the 1.17 pairs()/align ladder; demote dataset()/research() to legacy - README intro, quickstart, and 'what you can build' now lead with weather.pairs() + the three-rung align ladder (snippets verified executable offline against the shipped 1.17 surface); markets examples moved to kalshi/polymarket.pairs(); stale D-22 FutureWarning note removed (the flip was cancelled in 1.17); docs index promotes the migration guide and marks docs/dataset.md as the legacy composer doc. - docs/dataset.md gets a legacy-surface banner; docs/ts-quickstart.md gets a TS-on-1.17 note (research()/dataset() current THERE, ladder Python-only, PT-3401). - BYO-spine example documents the date join-key requirement (the raw-MergeError UX gap is filed separately). No code changes. Co-Authored-By: Claude Fable 5 --- README.md | 115 +++++++++++++++++++++++++++--------------- docs/dataset.md | 7 +++ docs/ts-quickstart.md | 5 ++ 3 files changed, 86 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 4065d045..1ab6e568 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![Docs](https://img.shields.io/badge/docs-mostlyright.md-blue.svg)](https://mostlyright.md/docs/sdk/) -`mostlyright` gives you one clean interface to public data sources — **general-first**. Today it ships weather (live observations, forecasts, climate history) as source-identified domain tables, plus `dataset()`, a leakage-safe training-table builder that aligns them to any label you choose. Prediction markets (Kalshi, Polymarket) are a thin convenience layer on top. Local-first, no hosted backend, no API key. +`mostlyright` gives you one clean interface to public data sources — **general-first**. Today it ships weather (live observations, forecasts, climate history) and economic indicators as source-identified domain tables, a per-domain `pairs()` quickstart, and `mr.align(spine, *sources)` — one leakage-guarded composition operator that as-of-joins any sources onto any label spine. Prediction markets (Kalshi, Polymarket) are a thin convenience layer on top. Local-first, no hosted backend, no API key. --- @@ -30,19 +30,33 @@ Python 3.11+. Node 18+. No API key required for any package below. ```python # Python -import mostlyright +from mostlyright import weather -# dataset() is the alignment engine; research() is a fully working alias. -# Pass an explicit label= (here the NWS CLI settlement extremes). -df = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12", label="cli") +# pairs() is the capped quickstart (v0.14.1 heritage: client.pairs()). +# label="cli" (the NWS CLI settlement extremes) is the documented default. +df = weather.pairs("KNYC", "2025-01-06", "2025-01-12") print(df.head()) # pandas DataFrame: one row per LST settlement date # Columns: date, station, cli_high_f, cli_low_f, obs_high_f, obs_low_f, -# obs_mean_f, obs_count, fcst_*, market_close_utc +# obs_mean_f, obs_count, ..., market_close_utc +``` + +When you outgrow the one-liner, you don't add kwargs — you graduate one rung +down to the same call it desugars to: + +```python +import mostlyright as mr +from mostlyright import weather + +spine = weather.label.cli("KNYC", "2025-01-06", "2025-01-12") # the target (y) +frame = mr.align(spine, weather.obs("KNYC"), weather.forecasts("KNYC")) +# align() as-of-joins each source onto the spine and enforces the leakage +# guard (source.knowledge_time <= spine.decision_time) in exactly one place. ``` ```ts -// TypeScript +// TypeScript — the TS SDK keeps research()/dataset() unchanged on 1.17 +// (align/spine/pairs are Python-only for now; parity ticket PT-3401). import { research } from "mostlyright"; const rows = await research("KNYC", "2025-01-06", "2025-01-12"); @@ -50,6 +64,11 @@ console.log(rows[0]); // ReadonlyArray: same schema as Python, JSON-serializable ``` +> **Coming from `dataset()` / `research()`?** Both still work and return the +> same bytes — they are deprecation shims now, removed at 2.0. The +> [1.17 migration guide](docs/migration/v117-align-spine-sources.md) maps all +> 26 old kwargs to their new homes. + First call writes a parquet (Python) or JSON-envelope (Node) cache to `~/.mostlyright/cache/`. Subsequent calls in the same window are local-only — no network. Full quickstart with concepts at . @@ -60,7 +79,7 @@ Full quickstart with concepts at . | Package | Description | Downloads | Status | |---|---|---|---| -| [`mostlyrightmd`](https://pypi.org/project/mostlyrightmd/) | Core types, schemas, validators, the `research()` join, and snapshot primitives. Imports as `mostlyright`. | [![monthly downloads](https://badgen.net/pypi/dm/mostlyrightmd?label=monthly)](https://pypistats.org/packages/mostlyrightmd) | stable | +| [`mostlyrightmd`](https://pypi.org/project/mostlyrightmd/) | Core types, schemas, validators, the `align()`/`spine()` composition layer, and snapshot primitives. Imports as `mostlyright`. | [![monthly downloads](https://badgen.net/pypi/dm/mostlyrightmd?label=monthly)](https://pypistats.org/packages/mostlyrightmd) | stable | | [`mostlyrightmd-weather`](https://pypi.org/project/mostlyrightmd-weather/) | Weather data fetchers — live METAR (AWC), ASOS archive (IEM), historical observations (GHCNh), and NWS climate text products (CLI). Direct public-API access. | [![monthly downloads](https://badgen.net/pypi/dm/mostlyrightmd-weather?label=monthly)](https://pypistats.org/packages/mostlyrightmd-weather) | stable | | [`mostlyrightmd-markets`](https://pypi.org/project/mostlyrightmd-markets/) | Prediction-market data — Kalshi NHIGH/NLOW weather-contract resolvers, Polymarket discovery + settlement, and Kalshi + Polymarket trade history. | [![monthly downloads](https://badgen.net/pypi/dm/mostlyrightmd-markets?label=monthly)](https://pypistats.org/packages/mostlyrightmd-markets) | stable | | `mostlyrightmd-edgar` | SEC filings (10-K, 10-Q, 8-K) — direct EDGAR full-text + facts access. | n/a | planned | @@ -89,7 +108,7 @@ Full quickstart with concepts at . The domain tables are the raw ingredients — each row carries a durable **source identity** so a model trained on one provider never silently reconciles against -another. Fetch them directly and join by hand, or feed them to `dataset()`. +another. Fetch them directly and join by hand, or hand them to `mr.align()`. ```python from mostlyright.weather import obs, climate @@ -98,40 +117,52 @@ from mostlyright.weather import obs, climate o = obs("KNYC", "2024-01-01", "2024-12-31", granularity="observation") c = climate("KNYC", "2024-01-01", "2024-12-31") # NWS CLI settlement labels -# Forecasts compose into dataset() via features=["forecasts"]; the raw NWP -# table is mostlyright.forecasts.forecast_nwp (needs the [nwp] extra). +# weather.forecasts(...) is the registered forecast source for align(); the +# raw NWP table is mostlyright.forecasts.forecast_nwp (needs the [nwp] extra). ``` -### Build a leakage-safe training table with `dataset()` +### Compose a leakage-safe training table — the `align` ladder -`dataset()` is the **alignment engine** — a training-table builder, not a fixed -recipe. Give it a station, a window, a **label axis**, and the features you want; -it returns one leakage-safe row per LST settlement day, every column aligned to the -settlement calendar and joined as-of so nothing from the future leaks backward. -`research()` is a fully working alias in this release. +Sources (features / X) and label spines (targets / y) are **orthogonal**; +`mr.align(spine, *sources)` is the one composition operator that joins them. +Every intermediate product is a plain, inspectable `pd.DataFrame`, and every +rung is supported — pick how much you want the SDK to do: ```python -import mostlyright - -# A general feature table — works for ANY catalog station worldwide, no market -# required. label=None composes features only (no settlement label columns). -df = mostlyright.dataset("EGLL", "2025-01-06", "2025-01-12", label=None) - -# Attach the WU/NOAA-WRH daily extremes as the label (native Celsius names). -df = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12", label="daily_extremes") - -# Or bring your own label frame — the SDK aligns it to the settlement calendar. -df = mostlyright.dataset("KNYC", "2025-01-06", "2025-01-12", label=my_label_frame) +import mostlyright as mr +from mostlyright import weather + +# Rung 3 — the per-domain quickstart one-liner (capped kwargs, never grows). +pairs = weather.pairs("KNYC", "2025-01-01", "2025-01-31") # label="cli" default + +# Rung 2 — spine + align: the SDK does ONLY the dangerous step (the as-of join). +spine = weather.label.daily_extremes("KNYC", "2025-01-01", "2025-01-31") +frame = mr.align(spine, weather.obs("KNYC"), weather.forecasts("KNYC")) + +# Rung 1 — raw source frames; build your own spine, or re-enter with mr.spine(). +raw_obs = weather.obs("KNYC", "2025-01-01", "2025-01-31") +# BYO label frame: keep a `date` column (the ISO settlement day) — it is the +# key weather sources attach on. mr.spine() maps YOUR column names explicitly. +spine = mr.spine(my_label_frame, + entity="station", decision_time="cutoff", y="settle_high") +frame = mr.align(spine, weather.obs("KNYC")) ``` -> Calling `dataset()` **without** an explicit `label=` still returns the NWS CLI -> settlement columns byte-identically, but emits a `FutureWarning`: the default -> flips to `label=None` at 2.0. Pass `label="cli"` or `label=None` to silence it. -> Full walkthrough in [`docs/dataset.md`](docs/dataset.md). +Features-only (no settlement label)? Align sources onto the bare settlement +calendar: `mr.align(weather.days("KSEA", d1, d2), weather.obs("KSEA"))`. + +> **Legacy surface.** `dataset()` / `research()` — the previous 26-kwarg +> composer — still work and return identical bytes, as deprecation shims +> removed at 2.0. No `FutureWarning` fires on the old default anymore (the +> planned label-default flip was cancelled). The +> [1.17 migration guide](docs/migration/v117-align-spine-sources.md) maps +> every old kwarg to its new home; the legacy walkthrough lives in +> [`docs/dataset.md`](docs/dataset.md). ### Train models with train/infer source parity -`dataset()` (and its `research()` alias) stamps a source identity on every row. +Every composed frame — `pairs()`, `align()`, and the legacy shims alike — +stamps a source identity on every row. Pinning `source=` on `obs()` returns single-source coverage; the validator catches train/infer mismatches at load time, instead of silently corrupting predictions in production. @@ -155,21 +186,22 @@ validate_dataframe(train, schema_id="schema.observation.v1") ### Backtest prediction markets (thin convenience layer) -The prediction-market entry points are thin delegators to the venue-free -`dataset()` — they own **all** venue knowledge (station resolution, strike parsing, -outcome targets) and route to the **venue-correct** label: Kalshi settles on NWS CLI +The prediction-market quickstarts are thin, venue-aware delegators — they own +**all** venue knowledge (station resolution, strike parsing, outcome targets) +and route to the **venue-correct** label: Kalshi settles on NWS CLI (`label="cli"`), Polymarket on WU/NOAA-WRH extremes (`label="daily_extremes"`). `outcome=True` appends a binary `label_outcome` by binarizing that venue's label -against its parsed strike (inclusive `>=`). +against its parsed strike (inclusive `>=`). Venue label spines +(`markets.kalshi.label.settlement(...)`) compose with `mr.align()` like any other. ```python from mostlyright.markets import kalshi, polymarket # Kalshi → NWS CLI settlement. outcome=True adds the binary YES/NO target. -k = kalshi.dataset("KXHIGHNY-25MAY26-T79", "2025-05-26", "2025-05-26", outcome=True) +k = kalshi.pairs("KXHIGHNY-25MAY26-T79", "2025-05-26", "2025-05-26", outcome=True) # Polymarket → WU/NOAA-WRH extremes (the correct ground truth — NOT cli_*). -p = polymarket.dataset("", "2025-05-26", "2025-05-26", outcome=True) +p = polymarket.pairs("", "2025-05-26", "2025-05-26", outcome=True) ``` ### Feed AI agents structured public data @@ -244,13 +276,14 @@ table, coverage map, auto-routing, cheap-CONUS steering, DSRF gating, the 28 TB - **No hosted backend.** Direct calls to public APIs (NOAA, NWS, IEM, Kalshi, Polymarket). No proxy. No vendor account. No rate-limited tier. - **Local-first cache.** Parquet (Python) or JSON envelope (Node) at `~/.mostlyright/cache/`. Byte-stable across runs — deterministic backtests. - **Schema-versioned outputs.** Every response carries a stable `schema.*.v1` URI. Train/infer source mismatches fail loudly instead of silently corrupting models. -- **Python + TypeScript peers.** Same `research()` shape, byte-equivalent on the parity fixtures. Use whichever runtime your stack prefers. +- **Python + TypeScript peers.** Same training-table shape (`weather.pairs()` in Python, `research()` in TS), byte-equivalent on the parity fixtures. Use whichever runtime your stack prefers. - **MIT licensed.** Use it commercially. Fork it. Ship it. ## Documentation - **Quickstart + concepts:** -- **`dataset()` — the alignment engine:** [`docs/dataset.md`](docs/dataset.md) — the label axis, the contributor protocol, panels, and the honest with/without comparison. +- **The `align` / `spine` / `sources` ladder (the 1.17 surface):** [`docs/migration/v117-align-spine-sources.md`](docs/migration/v117-align-spine-sources.md) — the three rungs, per-domain `pairs()`, and the composition model. +- **`dataset()` — the legacy composer (deprecated, removed at 2.0):** [`docs/dataset.md`](docs/dataset.md) — the label axis, the contributor protocol, panels; kept for shim users. - **Source identity — the cross-vertical kwarg contract:** [`docs/source-identity.md`](docs/source-identity.md) — `source=` / `delivery=` / grain, the labels-only firewall, and the fetch-provenance ledger. - **Migrating to 1.17.0 — the `align` / `spine` / `sources` ladder:** [`docs/migration/v117-align-spine-sources.md`](docs/migration/v117-align-spine-sources.md) — the where-did-each-`dataset()`-kwarg-go table (all 26), `label`'s 5 jobs → 4 call shapes, per-domain `pairs()`, the namespace-inversion symbol map, TypeScript-on-1.17, and the econ-unchanged note. Additive-plus-shims; the D-22 `FutureWarning` is gone. - **Migrating to 1.15.0:** [`docs/migration-1.15.md`](docs/migration-1.15.md) — the D-22 default-label `FutureWarning`, `include_*` → `features=`, and `contract=` → the markets layer. diff --git a/docs/dataset.md b/docs/dataset.md index 91eb434c..e3afe23a 100644 --- a/docs/dataset.md +++ b/docs/dataset.md @@ -1,5 +1,12 @@ # `dataset()` — the alignment engine +> **Legacy surface (1.17+).** `dataset()` and `research()` are deprecation +> shims as of 1.17.0 — they keep returning identical bytes but are removed at +> 2.0. The current surface is the per-domain `pairs()` quickstart and the +> `mr.align(spine, *sources)` ladder; see the +> [1.17 migration guide](migration/v117-align-spine-sources.md) for the +> complete kwarg-by-kwarg mapping. This document is kept for shim users. + `dataset()` is the one thing in mostlyright that is genuinely hard to get right by hand: it takes a station, a window, and a set of feature ingredients, and returns **one leakage-safe row per LST settlement day** — every column aligned to diff --git a/docs/ts-quickstart.md b/docs/ts-quickstart.md index 18c80136..3354e755 100644 --- a/docs/ts-quickstart.md +++ b/docs/ts-quickstart.md @@ -1,5 +1,10 @@ # TypeScript SDK quickstart (`@mostlyrightmd/*`) +> **TS on 1.17:** the TS SDK keeps `research()` / `dataset()` unchanged — +> they are current here, not deprecated. Python's new `align`/`spine`/`pairs()` +> ladder is Python-only for now (parity ticket PT-3401); see the +> ["TypeScript on 1.17" section of the migration guide](migration/v117-align-spine-sources.md). + The TS SDK mirrors the Python public surface and ships four npm packages: | npm | Path | Use case | From 91a183fff64174c8a3df69de9ec42ae28c9c74bc Mon Sep 17 00:00:00 2001 From: helloiamvu Date: Wed, 15 Jul 2026 10:11:54 +0200 Subject: [PATCH 2/4] docs(review-fixes): TS D-22 warning still ships, valid event-id placeholder, core-package description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex findings on 19c96cc: (1) the TS-on-1.17 banner overclaimed — TS dataset() still emits the D-22 FutureWarning on omitted label (the Python flip-cancellation was not ported; divergence filed separately); (2) '' fails _EVENT_ID_RE before any network call — placeholder is now a regex-valid slug; (3) @mostlyrightmd/core does not export research() (it lives in the mostlyright meta package). Co-Authored-By: Claude Fable 5 --- README.md | 10 ++++++---- docs/ts-quickstart.md | 10 +++++++--- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 1ab6e568..87e002f1 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ Full quickstart with concepts at . | Package | Description | Downloads | Status | |---|---|---|---| | [`mostlyright`](https://www.npmjs.com/package/mostlyright) | Meta package — one `import { research } from "mostlyright"` for weather data + prediction-market settlements + the core join. | [![monthly downloads](https://img.shields.io/npm/dm/mostlyright?label=monthly)](https://www.npmjs.com/package/mostlyright) | stable | -| [`@mostlyrightmd/core`](https://www.npmjs.com/package/@mostlyrightmd/core) | Core types, schemas, validators, temporal-safety primitives, and the `research()` join. | [![monthly downloads](https://img.shields.io/npm/dm/%40mostlyrightmd%2Fcore?label=monthly)](https://www.npmjs.com/package/@mostlyrightmd/core) | stable | +| [`@mostlyrightmd/core`](https://www.npmjs.com/package/@mostlyrightmd/core) | Core types, schemas, validators, and temporal-safety primitives (`research()` itself is exported by the `mostlyright` meta package). | [![monthly downloads](https://img.shields.io/npm/dm/%40mostlyrightmd%2Fcore?label=monthly)](https://www.npmjs.com/package/@mostlyrightmd/core) | stable | | [`@mostlyrightmd/weather`](https://www.npmjs.com/package/@mostlyrightmd/weather) | Weather data fetchers — live METAR (AWC), ASOS archive (IEM), historical observations (GHCNh), and NWS climate text products (CLI). | [![monthly downloads](https://img.shields.io/npm/dm/%40mostlyrightmd%2Fweather?label=monthly)](https://www.npmjs.com/package/@mostlyrightmd/weather) | stable | | [`@mostlyrightmd/markets`](https://www.npmjs.com/package/@mostlyrightmd/markets) | Prediction-market data — Kalshi NHIGH/NLOW weather-contract resolvers, Polymarket discovery + settlement, and Kalshi + Polymarket trade history. | [![monthly downloads](https://img.shields.io/npm/dm/%40mostlyrightmd%2Fmarkets?label=monthly)](https://www.npmjs.com/package/@mostlyrightmd/markets) | stable | | `@mostlyrightmd/edgar` | SEC filings (10-K, 10-Q, 8-K) — direct EDGAR full-text + facts access. | n/a | planned | @@ -153,8 +153,9 @@ calendar: `mr.align(weather.days("KSEA", d1, d2), weather.obs("KSEA"))`. > **Legacy surface.** `dataset()` / `research()` — the previous 26-kwarg > composer — still work and return identical bytes, as deprecation shims -> removed at 2.0. No `FutureWarning` fires on the old default anymore (the -> planned label-default flip was cancelled). The +> removed at 2.0. No `FutureWarning` fires on the old default anymore in +> Python (the planned label-default flip was cancelled; TS `dataset()` still +> emits it pending the ladder port). The > [1.17 migration guide](docs/migration/v117-align-spine-sources.md) maps > every old kwarg to its new home; the legacy walkthrough lives in > [`docs/dataset.md`](docs/dataset.md). @@ -201,7 +202,8 @@ from mostlyright.markets import kalshi, polymarket k = kalshi.pairs("KXHIGHNY-25MAY26-T79", "2025-05-26", "2025-05-26", outcome=True) # Polymarket → WU/NOAA-WRH extremes (the correct ground truth — NOT cli_*). -p = polymarket.pairs("", "2025-05-26", "2025-05-26", outcome=True) +# The event id comes from Polymarket discovery (alphanumeric slug). +p = polymarket.pairs("your-event-id", "2025-05-26", "2025-05-26", outcome=True) ``` ### Feed AI agents structured public data diff --git a/docs/ts-quickstart.md b/docs/ts-quickstart.md index 3354e755..d650b0fa 100644 --- a/docs/ts-quickstart.md +++ b/docs/ts-quickstart.md @@ -1,8 +1,12 @@ # TypeScript SDK quickstart (`@mostlyrightmd/*`) -> **TS on 1.17:** the TS SDK keeps `research()` / `dataset()` unchanged — -> they are current here, not deprecated. Python's new `align`/`spine`/`pairs()` -> ladder is Python-only for now (parity ticket PT-3401); see the +> **TS on 1.17:** the TS SDK surface is unchanged — `research()` and +> `dataset()` keep working exactly as in 1.16, including `dataset()`'s +> one-time D-22 `FutureWarning` on an omitted `label=` (pass `label` explicitly +> to silence it; the Python side cancelled that flip in 1.17 and the TS +> reconciliation is tracked with the ladder port). Python's new +> `align`/`spine`/`pairs()` ladder is Python-only for now (parity ticket +> PT-3401); see the > ["TypeScript on 1.17" section of the migration guide](migration/v117-align-spine-sources.md). The TS SDK mirrors the Python public surface and ships four npm packages: From b1dc2e6bac7ddc288b1d9c04a3ec9b5e7f48f573 Mon Sep 17 00:00:00 2001 From: minereda <84080887+minereda@users.noreply.github.com> Date: Thu, 16 Jul 2026 13:10:48 +0200 Subject: [PATCH 3/4] docs: correct 1.17 ladder and TS parity claims --- README.md | 9 +++++---- docs/dataset.md | 37 ++++++------------------------------- docs/ts-quickstart.md | 10 +++++----- 3 files changed, 16 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index 87e002f1..94e52375 100644 --- a/README.md +++ b/README.md @@ -37,8 +37,8 @@ from mostlyright import weather df = weather.pairs("KNYC", "2025-01-06", "2025-01-12") print(df.head()) # pandas DataFrame: one row per LST settlement date -# Columns: date, station, cli_high_f, cli_low_f, obs_high_f, obs_low_f, -# obs_mean_f, obs_count, ..., market_close_utc +# Columns include the settlement spine (`date`, `station`, `decision_time`, +# `label_available_time`) plus cli_* and obs_* values. ``` When you outgrow the one-liner, you don't add kwargs — you graduate one rung @@ -49,7 +49,7 @@ import mostlyright as mr from mostlyright import weather spine = weather.label.cli("KNYC", "2025-01-06", "2025-01-12") # the target (y) -frame = mr.align(spine, weather.obs("KNYC"), weather.forecasts("KNYC")) +frame = mr.align(spine, weather.obs("KNYC")) # align() as-of-joins each source onto the spine and enforces the leakage # guard (source.knowledge_time <= spine.decision_time) in exactly one place. ``` @@ -61,7 +61,8 @@ import { research } from "mostlyright"; const rows = await research("KNYC", "2025-01-06", "2025-01-12"); console.log(rows[0]); -// ReadonlyArray: same schema as Python, JSON-serializable +// ReadonlyArray: the legacy TS research() schema, JSON-serializable. +// PT-3401 tracks parity with Python's new spine/align output. ``` > **Coming from `dataset()` / `research()`?** Both still work and return the diff --git a/docs/dataset.md b/docs/dataset.md index e3afe23a..78232217 100644 --- a/docs/dataset.md +++ b/docs/dataset.md @@ -25,8 +25,7 @@ conveniences (`markets.kalshi.dataset` / `markets.polymarket.dataset`) are thin delegators that call this same `dataset()` — see [§ Markets are thin sugar](#markets-are-thin-sugar). > `research()` is a **fully working alias** of `dataset()` in this release. The -> two are distinct thin wrappers over one shared body; they behave identically -> except for one default-label warning (see the [D-22 bridge](#the-label-axis)). +> two are distinct thin wrappers over one shared body and behave identically. --- @@ -99,36 +98,12 @@ df = mostlyright.dataset("EGLL", "2025-01-06", "2025-01-12", label=None) # No cli_* label columns — just the composed, settlement-aligned features. ``` -### The D-22 bridge — the default label is changing +### Default label -Today, calling `dataset()` **without** an explicit `label=` still returns the -`cli_*` columns byte-identically — but it emits a one-time `FutureWarning`: - -> `dataset() label default changes from 'cli' to None in 2.0; pass label='cli' to -> keep settlement labels or label=None to silence` - -This is the **D-22 bridge**: a ≥2-minor deprecation window so no one is surprised -when the default flips to the pure `None` (features-only) table at 2.0. Pass an -explicit label and the warning never fires. - -| Release | `dataset()` with no `label=` | Warning | `research()` with no `label=` | -| --- | --- | --- | --- | -| 1.15 (now) | returns `cli_*` (byte-identical) | `FutureWarning` (once) | returns `cli_*`, **never warns** | -| 1.16 … 1.x | returns `cli_*` (byte-identical) | `FutureWarning` (once) | returns `cli_*`, **never warns** | -| 2.0 | default flips to `label=None` | — (default is now explicit) | *(legacy alias retires at 2.0)* | - -`research()` is the legacy name — it keeps the implicit `cli` label **forever with -no new warning** (it retires at 2.0 anyway, so there is nothing to migrate toward). -Parity tests pin `label="cli"` explicitly, so the parity gate stays warning-free by -construction. - -> **A note on `-W error`.** mostlyright's own CI and test suites do **not** run -> Python with warnings-as-errors — the D-22 `FutureWarning` is purely a -> **user-facing, opt-in concern**. If *you* run your code under `python -W error` -> (or a `filterwarnings = error` pytest config), the bare-`dataset()` call will -> raise instead of warn; pass an explicit `label="cli"` / `label=None` to keep it -> silent. Parity byte-stability is guaranteed by the explicit `label="cli"` pins, -> not by any warnings filter. +The proposed D-22 default flip was canceled. Calling `dataset()` or `research()` +without `label=` keeps the implicit `"cli"` label and emits no default-label +warning. Both legacy names retire at 2.0; use `weather.pairs()` for the capped +recipe or construct an explicit spine and sources with `align()`. ### Bring your own label diff --git a/docs/ts-quickstart.md b/docs/ts-quickstart.md index d650b0fa..fbc13b99 100644 --- a/docs/ts-quickstart.md +++ b/docs/ts-quickstart.md @@ -1,10 +1,8 @@ # TypeScript SDK quickstart (`@mostlyrightmd/*`) > **TS on 1.17:** the TS SDK surface is unchanged — `research()` and -> `dataset()` keep working exactly as in 1.16, including `dataset()`'s -> one-time D-22 `FutureWarning` on an omitted `label=` (pass `label` explicitly -> to silence it; the Python side cancelled that flip in 1.17 and the TS -> reconciliation is tracked with the ladder port). Python's new +> `dataset()` keep working exactly as in 1.16. The D-22 default-label flip and +> warning are canceled in both SDKs. Python's new > `align`/`spine`/`pairs()` ladder is Python-only for now (parity ticket > PT-3401); see the > ["TypeScript on 1.17" section of the migration guide](migration/v117-align-spine-sources.md). @@ -45,7 +43,9 @@ console.log(rows[0]); // } ``` -`research(station, fromDate, toDate)` returns the same 20-column shape Python emits — byte-equivalent on the 5 canonical parity fixtures. +`research(station, fromDate, toDate)` returns the legacy TS `PairsRow` shape. +PT-3401 tracks parity with Python's new spine/align output; the historical +five-fixture research contract remains pinned separately. ## Cache From 2e516112a26d518bd36cb34b3afa4e80fce3aaf2 Mon Sep 17 00:00:00 2001 From: minereda <84080887+minereda@users.noreply.github.com> Date: Thu, 16 Jul 2026 13:15:00 +0200 Subject: [PATCH 4/4] docs: correct provenance and market outcome contracts --- README.md | 15 ++++++++++----- docs/dataset.md | 2 +- docs/ts-quickstart.md | 4 ++-- 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 94e52375..b26ab1bd 100644 --- a/README.md +++ b/README.md @@ -164,7 +164,8 @@ calendar: `mr.align(weather.days("KSEA", d1, d2), weather.obs("KSEA"))`. ### Train models with train/infer source parity Every composed frame — `pairs()`, `align()`, and the legacy shims alike — -stamps a source identity on every row. +records frame/source provenance in `DataFrame.attrs`; raw source tables also +carry their schema-defined row-level source fields. Pinning `source=` on `obs()` returns single-source coverage; the validator catches train/infer mismatches at load time, instead of silently corrupting predictions in production. @@ -193,7 +194,9 @@ The prediction-market quickstarts are thin, venue-aware delegators — they own and route to the **venue-correct** label: Kalshi settles on NWS CLI (`label="cli"`), Polymarket on WU/NOAA-WRH extremes (`label="daily_extremes"`). `outcome=True` appends a binary `label_outcome` by binarizing that venue's label -against its parsed strike (inclusive `>=`). Venue label spines +against its parsed strike. Threshold/range direction follows venue terms: +Kalshi high thresholds use inclusive `>=`; Polymarket below markets use `<=`, +and between markets use inclusive bounds. Venue label spines (`markets.kalshi.label.settlement(...)`) compose with `mr.align()` like any other. ```python @@ -203,7 +206,7 @@ from mostlyright.markets import kalshi, polymarket k = kalshi.pairs("KXHIGHNY-25MAY26-T79", "2025-05-26", "2025-05-26", outcome=True) # Polymarket → WU/NOAA-WRH extremes (the correct ground truth — NOT cli_*). -# The event id comes from Polymarket discovery (alphanumeric slug). +# The event id comes from the discovery row's `event_id` field (not `slug`). p = polymarket.pairs("your-event-id", "2025-05-26", "2025-05-26", outcome=True) ``` @@ -279,7 +282,9 @@ table, coverage map, auto-routing, cheap-CONUS steering, DSRF gating, the 28 TB - **No hosted backend.** Direct calls to public APIs (NOAA, NWS, IEM, Kalshi, Polymarket). No proxy. No vendor account. No rate-limited tier. - **Local-first cache.** Parquet (Python) or JSON envelope (Node) at `~/.mostlyright/cache/`. Byte-stable across runs — deterministic backtests. - **Schema-versioned outputs.** Every response carries a stable `schema.*.v1` URI. Train/infer source mismatches fail loudly instead of silently corrupting models. -- **Python + TypeScript peers.** Same training-table shape (`weather.pairs()` in Python, `research()` in TS), byte-equivalent on the parity fixtures. Use whichever runtime your stack prefers. +- **Python + TypeScript peers.** Python 1.17 uses the spine/align shape; TS + `research()` retains the legacy `PairsRow` shape while PT-3401 tracks the + ladder port. Use whichever runtime your stack prefers. - **MIT licensed.** Use it commercially. Fork it. Ship it. ## Documentation @@ -288,7 +293,7 @@ table, coverage map, auto-routing, cheap-CONUS steering, DSRF gating, the 28 TB - **The `align` / `spine` / `sources` ladder (the 1.17 surface):** [`docs/migration/v117-align-spine-sources.md`](docs/migration/v117-align-spine-sources.md) — the three rungs, per-domain `pairs()`, and the composition model. - **`dataset()` — the legacy composer (deprecated, removed at 2.0):** [`docs/dataset.md`](docs/dataset.md) — the label axis, the contributor protocol, panels; kept for shim users. - **Source identity — the cross-vertical kwarg contract:** [`docs/source-identity.md`](docs/source-identity.md) — `source=` / `delivery=` / grain, the labels-only firewall, and the fetch-provenance ledger. -- **Migrating to 1.17.0 — the `align` / `spine` / `sources` ladder:** [`docs/migration/v117-align-spine-sources.md`](docs/migration/v117-align-spine-sources.md) — the where-did-each-`dataset()`-kwarg-go table (all 26), `label`'s 5 jobs → 4 call shapes, per-domain `pairs()`, the namespace-inversion symbol map, TypeScript-on-1.17, and the econ-unchanged note. Additive-plus-shims; the D-22 `FutureWarning` is gone. +- **Migrating to 1.17.0 — the `align` / `spine` / `sources` ladder:** [`docs/migration/v117-align-spine-sources.md`](docs/migration/v117-align-spine-sources.md) — the where-did-each-`dataset()`-kwarg-go table (all 26), `label`'s 5 jobs → 4 call shapes, per-domain `pairs()`, the namespace-inversion symbol map, TypeScript-on-1.17, and the econ-unchanged note. Additive-plus-shims; Python removed the D-22 `FutureWarning`, with the paired TS cancellation in #127. - **Migrating to 1.15.0:** [`docs/migration-1.15.md`](docs/migration-1.15.md) — the D-22 default-label `FutureWarning`, `include_*` → `features=`, and `contract=` → the markets layer. - **API reference:** (Python + TypeScript reference auto-generated per release) - **Migration from the legacy hosted-API client:** diff --git a/docs/dataset.md b/docs/dataset.md index 78232217..7ceed9b2 100644 --- a/docs/dataset.md +++ b/docs/dataset.md @@ -44,7 +44,7 @@ byte-stable primitive. Here is what each capability costs you by hand versus wha | One row per settlement day | You dedup + aggregate + guard against fan-out | Row count is unique-per-day by construction | | Attach a label | You pick a label source, join it, null low-coverage days | `label=` routes a named recipe / `None` / a BYO frame | | Multi-station panel | You loop, concat, re-key, preserve order | `stations=[...]` — long format, order preserved | -| Source identity on every row | You track which provider each row came from | Durable per-row `source` column + frame identity | +| Source provenance | You track which provider each input came from | Frame/source identity in `DataFrame.attrs`; raw tables retain schema-defined row source fields | `dataset()` is worth reaching for exactly when the day-alignment and the no-leakage join are the parts you'd otherwise get subtly wrong. If you only need diff --git a/docs/ts-quickstart.md b/docs/ts-quickstart.md index fbc13b99..70d92549 100644 --- a/docs/ts-quickstart.md +++ b/docs/ts-quickstart.md @@ -1,8 +1,8 @@ # TypeScript SDK quickstart (`@mostlyrightmd/*`) > **TS on 1.17:** the TS SDK surface is unchanged — `research()` and -> `dataset()` keep working exactly as in 1.16. The D-22 default-label flip and -> warning are canceled in both SDKs. Python's new +> `dataset()` keep working exactly as in 1.16. The paired TS cancellation in +> PR #127 removes the D-22 default-label flip and warning. Python's new > `align`/`spine`/`pairs()` ladder is Python-only for now (parity ticket > PT-3401); see the > ["TypeScript on 1.17" section of the migration guide](migration/v117-align-spine-sources.md).