test(parity): pinned-input HARD gate (deterministic, fast-suite) + live drift monitor#124
Merged
Conversation
… + live drift monitor The live parity gate compared today's code + today's MUTABLE IEM/AWC archive against fixtures frozen from mostlyright==0.14.1 in May 2026 — so upstream restatements (late/corrected METARs) failed the gate on code that is provably byte-identical, and the 0.14.1 oracle can never be re-captured (api.mostlyright.md re-pointed in Phase 28). Redesign (operator-directed): - tests/fixtures/parity/pinned_cache/ — committed input bundle (25 files, 1.4MB): the SDK's own derived cache tiers (v1/observations monthly + v1/climate yearly parquets) for all 5 windows, captured live 2026-07-15 from main @ 02580c3 by the new capture_pinned_cache.py. Raw source text + provenance excluded (replay verified byte-identical without them). - test_parity_case_pinned — the HARD gate: research() fully OFFLINE over the pinned bundle with a socket-level network block (an incomplete bundle fails loudly, never silently goes live). Deterministic forever; runs in the DEFAULT fast suite (2s for all 5 cases) so parity is now checked on every CI push, not just pre-publish. - test_parity_case_live — the old live run, kept under -m live as an explicit upstream-drift monitor with a documented audit + re-pin procedure (README). - Fixture cell re-base: 55 cells across case-3 (1 KLAX day) + case-4 (17 KMIA days) — obs_mean_f / obs_mean_dewpoint_f / obs_total_precip_in / obs_count ONLY, every diff count-consistent — re-based onto the pinned archive; per-cell audit table in fixtures/parity/README.md. Settlement-critical columns (cli_*, obs_high_f/obs_low_f) untouched on all cases, and cases 1/2/5 reproduce 0.14.1 EXACTLY from the pinned inputs (validating that current code is byte-faithful to 0.14.1 wherever the archive did not move). ts/case_3+4 JSON + manifest regenerated via export_for_ts.py. Precedent: the 2026-07-09 case-4 re-baseline (README). Gate results: pinned 5/5 byte+dtype green (2.1s); live monitor 5/5 green post-re-base; full fast suite 4616 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Parity ticket gate: PASSED See |
…rs + merge in the gate Codex PR-124 CRITICAL: pinning the derived v1/observations + v1/climate parquet tiers let the warm-cache path skip the raw IEM/AWC/GHCNh/CLI parsers and merge_observations/merge_climate — a parser or SOURCE_PRIORITY regression would have passed the hard gate. (The Python Architect independently rated the old design an acceptable PASS given dedicated parser/merge units + the -m live monitor; this pivot satisfies both verdicts by strictly widening gate coverage.) - The bundle is now tests/fixtures/parity/pinned_cache.tar.xz — RAW v1/sources fetch text only (82 MB raw -> 5.1 MB xz), no derived tiers. - test_parity_case_pinned replays CACHE-COLD: production parse -> merge -> aggregate -> join all execute under the socket block on every run; derived tiers are rebuilt into a per-module tmp root (within a case the 2nd/3rd research() calls go warm — production cache behavior; the 5 cases share no station). All-5-case cold replay verified byte-identical to the warm-tier replay and to the re-based fixtures; 14/14 in ~7s. - capture_pinned_cache.py emits the tarball; module-scope unpack fixture (Python 3.11 tarfile filter= compat); check-added-large-files hook gets a single-path exclude for the 5.1 MB bundle. Fast suite 4616 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pinned gate is the FIRST time research() OUTPUT dtypes run under the
pandas-3 CI suite (the live gate was always skipped there). pd3 infers
datetime64[us] for constructed frames while pd2-written fixture parquets
read back as [ns] — the documented lossless shift from coerce_pd3.py,
whose parity wiring was designed in Phase 6 but never activated. Applied
inline: expected frames get the ns->us datetime half of the bridge (the
object->string half is NOT applied — pd3's parquet reader already lands
strings on 'str'); the expected-dtype translation is conditional on the
actual frame exactly like the existing string translation, so parquet-
read comparisons (test_dtypes_match_ground_truth) keep ns and only
{ns,us} are accepted — any other dtype still hard-fails.
Verified: tests/test_parity.py + tests/test_coerce_pd3.py 24/24 green
under BOTH pandas 2 (lockfile) and pandas 3 (--with 'pandas>=3.0,<4').
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Redesigns the 5-fixture parity gate so it tests our code, not the stability of IEM's mutable archive. Unblocks the v1.17.0 release gate (was 3/5 live from environmental
obs_mean_fdrift on KLAX/KMIA).test_parity_case_pinned— the HARD gate.research()runs fully offline (socket-level network block) over a committed input bundletests/fixtures/parity/pinned_cache/— the SDK's own derived cache tiers for the 5 windows, captured live 2026-07-15 by the newcapture_pinned_cache.py. Deterministic forever, 2.1s for all 5 cases, and lives in the default fast suite — parity now gates every CI push instead of only pre-publish.test_parity_case_live— upstream-drift monitor (kept under-m live). A failure while the pinned gate is green = archive drift, with a documented audit + deliberate re-pin procedure in the fixtures README.obs_mean_f/obs_mean_dewpoint_f/obs_total_precip_in/obs_count, every diff count-consistent (late/corrected METARs). Full per-cell audit table intests/fixtures/parity/README.md. Settlement-critical columns (cli_*,obs_high_f/obs_low_f) untouched on all cases. Precedent: the operator-approved 2026-07-09 case-4 re-baseline.ts/case_3+4, manifest) regenerated via the existingexport_for_ts.py.Gate results
Why not just re-capture fixtures?
The 0.14.1 oracle is gone (
api.mostlyright.mdre-pointed in Phase 28) — the May-2026 fixtures are the last ground truth. This change keeps every non-drifted cell at original 0.14.1 bytes and freezes the inputs so the comparison stays meaningful forever.Unblocks: v1.17.0 tag (Phase 34, merged in #123).
🤖 Generated with Claude Code