diff --git a/CHANGELOG.md b/CHANGELOG.md index a9081ba5..13c76070 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 purpose-phrasing, restoring the case as a genuinely clean negative control. ### Added +- **Internal: 4.0 diagnostic-family amendment to the design spec.** `docs/v4-design.md` + gains section 3.5 formalizing a third object kind - diagnostics: the library's + existing "Diagnostics & Sensitivity" family (parallel-trends testing, placebo + suite, HonestDiD, pre-trends power, power analysis) made checkable, with + `BaconDecomposition` and `RDPlot` re-homed into it at 4.0 and the upcoming + density test born into it. One load-bearing bit: estimator results carry the + canonical inference quintet, diagnostic results do not (exempt BY TYPE via a + single `Diagnostic` marker base on the diagnostic RESULT containers, shipping + additively in Phase 2 with a ledger-gated roster test (M-091) - zero new + classes beyond the base, no import-path moves; raw-dict diagnostic functions + participate in the docs family only). The 4.0 docs/roster + reorganization is ledger-tracked (M-090) so the release cut cannot miss it, + and `RDPlot.fit()`'s post-freeze `outcome_col`/`running_col` params get their + standard rename rows (M-088/M-089) under the spec's missed-rename clause. + Docs + ledger + test-count updates only - **no public API or numerical behavior + change.** - **Reviewer-eval harness: N-arm matrix, blinded grading, corpus grown 2 -> 11 (`tools/reviewer-eval/`, prep for the GPT-5.6 reviewer evaluation).** `config/configs.json` moves from the two-arm `control`/`candidate` shape to an diff --git a/docs/v4-deprecations.yaml b/docs/v4-deprecations.yaml index 1660c329..904d2388 100644 --- a/docs/v4-deprecations.yaml +++ b/docs/v4-deprecations.yaml @@ -869,6 +869,56 @@ rows: code_refs: [diff_diff/wooldridge_results.py] notes: "summary() unifies to the library-wide summary(alpha=None) signature (spec section 5); aggregation selection lives on aggregate(). The alpha param arrives additively in the same PR." + - id: M-088 + kind: param + group: renames-col-suffix + old: "diff_diff:RDPlot.fit[outcome_col]" + new: "diff_diff:RDPlot.fit[outcome]" + introduced_in: "3.9" + deprecated_in: "3.9" + removed_in: "4.0" + status: planned + phase: 2 + warning: FutureWarning + code_refs: [diff_diff/rdplot.py] + notes: "RDPlot shipped post-ledger-freeze with _col suffixes; the spec section 8 missed-rename clause applies (matches RegressionDiscontinuity [M-040])." + - id: M-089 + kind: param + group: renames-col-suffix + old: "diff_diff:RDPlot.fit[running_col]" + new: "diff_diff:RDPlot.fit[running]" + introduced_in: "3.9" + deprecated_in: "3.9" + removed_in: "4.0" + status: planned + phase: 2 + warning: FutureWarning + code_refs: [diff_diff/rdplot.py] + notes: "Companion to [M-088]; matches [M-041]." + - id: M-090 + kind: behavior + group: diagnostic-family + old: "diff_diff:BaconDecomposition" + new: null + deprecated_in: "4.0" + removed_in: null + status: planned + phase: 5 + code_refs: [README.md, diff_diff/guides/llms.txt, docs/api/index.rst, docs/index.rst, diff_diff/bacon.py] + notes: "4.0 diagnostic-family docs/roster reorganization (spec section 3.5): Bacon re-homed OUT of the API estimator roster (misfiled today); RDPlot (already documented as a diagnostic) consolidated under the unified family grouping; README/llms.txt/API/alias groupings (incl. docs/api/index.rst AND the docs homepage docs/index.rst, both listing Bacon as an estimator today) and estimator-count claims split into estimators + diagnostics; the flip's terminal test asserts the estimator/diagnostic grouping on the named surfaces. The Phase 2 marker introduction is gated separately by [M-091]." + - id: M-091 + kind: behavior + group: diagnostic-family + old: "diff_diff:BaconDecompositionResults" + new: null + introduced_in: "3.9" + deprecated_in: null + removed_in: null + status: planned + phase: 2 + code_refs: [diff_diff/bacon.py, diff_diff/rdplot.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/power.py, diff_diff/diagnostics.py, diff_diff/had_pretests.py, diff_diff/diagnostic_report.py, diff_diff/business_report.py, diff_diff/practitioner.py, diff_diff/_reporting_helpers.py, diff_diff/__init__.py] + notes: "Phase 2 Diagnostic marker base on the RESULT roster (spec section 3.5): BaconDecompositionResults, RDPlotResult, HonestDiDResults + SensitivityResults, PreTrendsPowerResults + PreTrendsPowerCurve, PowerResults + Simulation*Results, PlaceboTestResults, and the HAD pretest containers (QUGTestResults, StuteTestResults, YatchewTestResults, StuteJointResult, HADPretestReport), and DiagnosticReportResults (serialization pair moves/delegates from the builder onto the container). Flip to done ALSO requires consumer-propagation tests: BusinessReport rejects marked diagnostics as primary estimator input BY TYPE (not by name); practitioner_next_steps routes marked diagnostics through diagnostic handling (not the unknown-result estimator fallback); and DiagnosticReport routes by the marker - at least one non-Bacon marked result handled as a diagnostic, never via estimator fallback, with Bacon's existing read-out retained and tested. Flip to done requires the dedicated roster test (isinstance + summary/to_dataframe on every member; estimator results NOT marked). Raw-dict functions and TWFEWeightsResult are docs-family only (narrowed, spec section 3.5). introduced_in gates the 3.9 cut: the marker cannot be silently skipped." + # ---- Behavior policies (schema-tracked, spec-governed; no reality probe) - - id: M-080 kind: behavior diff --git a/docs/v4-design.md b/docs/v4-design.md index 74eddbac..1e81b2cf 100644 --- a/docs/v4-design.md +++ b/docs/v4-design.md @@ -91,7 +91,7 @@ Derived from `diff_diff.__all__` at v3.8.0. 24 estimator classes -> 21. | ChangesInChanges | Keep + absorbs QDiD via `method=` [M-015] | | QDiD | Removed 4.0 [M-015] | | SyntheticDiD, SyntheticControl, TROP, ContinuousDiD, HeterogeneousAdoptionDiD, RegressionDiscontinuity, SpilloverDiD, ChaisemartinDHaultfoeuille | Keep | -| BaconDecomposition | Keep (documented as a diagnostic, not an estimator, in the API docs and alias-table grouping) | +| BaconDecomposition | Keep - moves into the diagnostic family (section 3.5) | ### 3.2 Final alias table @@ -148,6 +148,86 @@ signatures. Lifecycle facts live in the cited rows. [M-045]..[M-047] - fully redundant with `vcov_type`, and its default even differed across estimators (True/True/False). +### 3.5 The diagnostic family + +diff-diff 4.0 formalizes a THIRD object kind alongside estimators and +results: **diagnostics** (locked with the maintainer 2026-07-19). Exactly +ONE bit is load-bearing: an estimator's result carries a causal-effect +inference row (the section 5 quintet); a diagnostic's result does NOT - +it assesses a design, an identifying assumption, or robustness (a +decomposition table, binned plot data, a pre-trends p-value, sensitivity +bounds, a power curve). + +**Membership: a NEW CANONICAL CONSOLIDATION anchored on the +"Diagnostics & Sensitivity" docs family** (which today is close but not +identical - the consolidation is the source of truth from 4.0): +BaconDecomposition (reclassified OUT of the README/API estimator lists), +RDPlot (consolidated from the RD grouping), parallel-trends testing, the +placebo suite, HonestDiD, PreTrendsPower, PowerAnalysis, +`twowayfeweights`, the HAD pretests, DiagnosticReport's results, and the +upcoming CJM 2020 density test (born into the family). Explicit +NON-members despite current docs placement: Conley spatial-HAC (an +embedded inference method on estimators, not a diagnostic object) and +other design elements listed under the README section. (A narrower +classes-only tier with separate categories for HonestDiD/PowerAnalysis +was drafted and REJECTED 2026-07-19: the boundary needed adjudication, +and the family line above is the decided one.) + +**Mechanics - the marker lives on RESULT containers.** Consumers hold +results, not entry points, so the `Diagnostic` marker base (shipping +ADDITIVELY in Phase 2 [M-091]) attaches to the class-backed diagnostic +RESULT types: `BaconDecompositionResults`, `RDPlotResult`, +`HonestDiDResults` + `SensitivityResults`, `PreTrendsPowerResults` + +`PreTrendsPowerCurve`, `PowerResults` + the `Simulation*Results` family, +`PlaceboTestResults`, the HAD pretest containers (`QUGTestResults`, +`StuteTestResults`, `YatchewTestResults`, `StuteJointResult`, +`HADPretestReport` - all already exposing the serialization pair), and +`DiagnosticReportResults` (whose `summary()`/`to_dataframe()` live on the +BUILDER today - Phase 2 moves or delegates them onto the container to meet +the contract; BusinessReport exports no result container, so nothing to +mark there) - plus every future diagnostic's result type (the density +test's first). Entry-point classes are NOT marked (one +marker, one meaning). Contract: marked results expose `summary()` / +`to_dataframe()` and are exempt from the section 5 quintet BY TYPE +(isinstance-checkable, so BusinessReport / DiagnosticReport / +practitioner routing stop special-casing by result-class name). The +family's entry points keep their existing shapes - data-in (Bacon, +RDPlot) and results-in (HonestDiD, PreTrendsPower) both belong; the +marker does not constrain them. NARROWING: function-shaped members that +return raw dicts (`check_parallel_trends` and variants, +`equivalence_test_trends`) and `TWFEWeightsResult` (no +`summary()`/`to_dataframe()` today) participate in the DOCS family only, +not the type contract; upgrading them to marked containers is optional +Phase 2 follow-up work, not part of this contract. The Phase 2 gate +[M-091] requires a dedicated roster test AND consumer propagation: every +enumerated result type is `isinstance(result, Diagnostic)` and exposes +the serialization pair, representative ESTIMATOR results are NOT, and the +CONSUMERS actually switch to the marker - BusinessReport rejects marked +diagnostics as its primary estimator input by type (today it +special-cases only Bacon by name), `practitioner_next_steps()` routes +marked diagnostics through diagnostic-specific handling instead of its +unknown-result estimator fallback, and DiagnosticReport itself routes by +the marker: at least one non-Bacon marked result is handled as a +diagnostic (never through estimator fallback) while Bacon's existing +read-out behavior is retained and tested. Import paths do NOT move - the +flat top-level namespace is kept (rejected: `diff_diff.diagnostics.*` +moves). Zero new classes beyond the single marker. + +**4.0 reorganization [M-090].** At 4.0 the public surface claims split +into "estimators + diagnostics": Bacon is re-homed OUT of the API +estimator roster (where it is misfiled today); RDPlot - already +documented under Diagnostics & Sensitivity - is consolidated under the +unified family grouping; README catalog, llms.txt, API reference and +alias-table groupings, and estimator-count claims on JOSS-adjacent +surfaces all split accordingly. Tracked as a behavior row so the cut +sweep cannot miss it. + +**RDPlot contract compliance [M-088] [M-089].** `RDPlot.fit()` shipped +with `outcome_col` / `running_col` after the ledger froze - the section 8 +missed-rename clause applies, and the params migrate to bare `outcome` / +`running` on the standard 3.9-shim / 4.0-removal path (matching +RegressionDiscontinuity's [M-040] [M-041]). + ## 4. The three merges ### 4.1 TwoWayFixedEffects absorbs MultiPeriodDiD [M-010] @@ -272,9 +352,11 @@ removed [M-015] [M-061]. **Canonical quintet.** `att`, `se`, `t_stat`, `p_value`, `conf_int` - bound to ONE coherent inference row (locked library principle: uniform names never -carry altered meaning). As of v3.8.0 every results class already exposes the -full quintet as properties (verified 2026-07-18), so 3.9 needs no additive -property work. At 4.0 the STORAGE flips [M-050]..[M-058]: the canonical names +carry altered meaning). The quintet contract applies to ESTIMATOR results; +diagnostic-family results (section 3.5) are exempt BY TYPE - they expose +`summary()` / `to_dataframe()` but no inference row. As of v3.8.0 every +ESTIMATOR result class already exposes the full quintet as properties +(verified 2026-07-18), so 3.9 needs no additive property work. At 4.0 the STORAGE flips [M-050]..[M-058]: the canonical names become the native dataclass fields; `overall_att` (and sibling `overall_*` inference fields, plus ContinuousDiD's deviant `overall_att_*` family [M-058]) become FutureWarning properties, removed 5.0. MultiPeriodDiD's @@ -409,10 +491,10 @@ above; anything only one PR cares about stays in that PR's plan.** | Phase | Ships in | PRs (each: dedicated shim/removal tests + matrix flips + CHANGELOG naming flipped row ids) | |---|---|---| | 1 (this PR) | - | Spec + matrix + enforcement test + support edits | -| 2: contract foundations | 3.9 | (a) results base + unified event-study representation + to_dict completion; (b) `aggregate()` + fit(aggregate=) shims [M-020..M-027]; (c) param renames [M-030..M-047] + BaseEstimator mixin + ContinuousDiD covariates move; (d) alias introductions [M-062] [M-063] + wrapper deprecations [M-070..M-077] + n_bootstrap docs [M-081] | +| 2: contract foundations | 3.9 | (a) results base + unified event-study representation + to_dict completion + the Diagnostic marker base on the diagnostic result roster [M-091] (section 3.5); (b) `aggregate()` + fit(aggregate=) shims [M-020..M-027]; (c) param renames [M-030..M-047] + BaseEstimator mixin + ContinuousDiD covariates move; (d) alias introductions [M-062] [M-063] + wrapper deprecations [M-070..M-077] + n_bootstrap docs [M-081] | | 3: merges | 3.9 | (a) TWFE event-study mode [M-010] + EventStudy warn [M-060] (gates: section 4.1's equivalence/divergence/pooled-parity test triple); (b) TripleDifference facade [M-013]; (c) CiC method= [M-015] | | 4: release + soak | 3.9 cut | Migration guide written (skeleton: section 10); maintainer cuts 3.9; maint/3.8 rule active | -| 5: enforcement | 4.0 | Removals [M-010..M-016, M-030..M-047 old names, M-060, M-061, M-070..M-077, M-001..M-003]; storage flips [M-050..M-058]; default policies [M-004..M-006, M-080]; warning retirement [M-007]; fastpath go/no-go [M-008]; docs/llms.txt/README refresh | +| 5: enforcement | 4.0 | Removals [M-010..M-016, M-030..M-047 old names, M-060, M-061, M-070..M-077, M-001..M-003]; storage flips [M-050..M-058]; default policies [M-004..M-006, M-080]; warning retirement [M-007]; fastpath go/no-go [M-008]; diagnostic-family docs/roster reorganization [M-090]; docs/llms.txt/README refresh | | 6: front door | 4.1 | `event_study(data, outcome, unit, time, first_treat, estimator=...)` comparison entry point over the staggered family (sketch only; specified in its own plan) | **4.0-cut checklist (final item):** the due-row sweep is AUTOMATED - @@ -505,8 +587,9 @@ forever - a removed symbol resurrecting is a test failure. `removed_in` (symbol/alias rows), flip version (`deprecated_in` on default-flip/warning-retirement/behavior rows), or `decision_due` (env-default rows), the status must be terminal; a due `introduced_in` - means the row may no longer be `planned` (the new surface must have - shipped - this is what gates introduce-only aliases); symbol rows with a + means the row may no longer be `planned` OR `evaluate` (the new surface + must have shipped; evaluate cannot satisfy an introduction - this is what + gates introduce-only aliases and the Phase 2 marker); symbol rows with a due `deprecated_in` and a declared `warning` may no longer be `planned`. The gate is two-sided: an EARLY-removal guard fails any row that goes terminal while its scheduled version is still in the future (the shim @@ -527,7 +610,7 @@ forever - a removed symbol resurrecting is a test failure. the same object as its target, so the deprecation warning rides the parent class row (schema-enforced). Top-level `diff_diff:Name` class/function rows and alias rows also assert `__all__` membership consistent with their - status (stale `import *` entries fail). The initial 71 row ids are a + status (stale `import *` entries fail). The initial 75 row ids are a committed snapshot in the enforcement test: ids are never deleted or reused, and the test fails if any snapshot id disappears. diff --git a/tests/test_v4_matrix.py b/tests/test_v4_matrix.py index 4d823d12..7d490790 100644 --- a/tests/test_v4_matrix.py +++ b/tests/test_v4_matrix.py @@ -109,14 +109,14 @@ _FIELD_RE = re.compile(r"^ ([a-z_]+):\s*(.*?)\s*$") _MD_TOKEN_RE = re.compile(r"\[(M-\d{3})\]") -# Row-count floor: exactly the 71 rows shipped at Phase 1. Ids are never reused and terminal +# Row-count floor: exactly the 75 rows shipped by the Phase 1 spec (incl. the diagnostic-family amendment). Ids are never reused and terminal # rows are never deleted, so the ledger only grows - raise the floor when rows are added; a # lower parse count means scanner/format drift or an illegal row deletion. -ROW_COUNT_FLOOR = 71 +ROW_COUNT_FLOOR = 75 # Committed snapshot of the Phase 1 id set ("ids are never deleted or reused" contract - a # delete-one-add-one edit keeps the count above the floor but trips this). Extend, never edit. -_INITIAL_ID_RANGES = [(1, 8), (10, 16), (20, 27), (30, 47), (50, 58), (60, 64), (70, 77), (80, 87)] +_INITIAL_ID_RANGES = [(1, 8), (10, 16), (20, 27), (30, 47), (50, 58), (60, 64), (70, 77), (80, 91)] EXPECTED_INITIAL_IDS = frozenset( f"M-{n:03d}" for lo, hi in _INITIAL_ID_RANGES for n in range(lo, hi + 1) ) @@ -427,8 +427,9 @@ def collect_due_problems(rows, current): - default-flip / warning-retirement / behavior rows past their flip version (``deprecated_in``) must be ``done``; - env-default rows past ``decision_due`` must be ``done`` (go/no-go recorded either way); - - ANY row past ``introduced_in`` may no longer be ``planned`` (the new surface must have - shipped - this is what gates introduce-only aliases); + - ANY row past ``introduced_in`` may no longer be ``planned`` OR ``evaluate`` (the new + surface must have shipped - this is what gates introduce-only aliases and the Phase 2 + marker; evaluate cannot satisfy an introduction); - symbol rows that declare a ``warning`` and are past ``deprecated_in`` must no longer be ``planned`` (their shim must have shipped); - EARLY-REMOVAL GUARD: a row must NOT be terminal while its scheduled version @@ -467,10 +468,18 @@ def collect_due_problems(rows, current): "(record the go/no-go either way)" ) introduced_in = row.get("introduced_in") - if introduced_in and current >= _version_tuple(introduced_in) and status == "planned": + if ( + introduced_in + and current >= _version_tuple(introduced_in) + and status + in ( + "planned", + "evaluate", + ) + ): problems.append( - f"{rid}: introduced_in {introduced_in} is due but status is still 'planned' " - "(new surface not shipped?)" + f"{rid}: introduced_in {introduced_in} is due but status is '{status}' - the " + "new surface must have shipped (evaluate cannot satisfy an introduction)" ) if ( kind in LIFECYCLE_KINDS @@ -505,7 +514,7 @@ def test_initial_ids_never_deleted(): ROW_COUNT_FLOOR alone would let a delete-one-add-one edit pass; this snapshot cannot.""" missing = sorted(EXPECTED_INITIAL_IDS - set(_ROW_IDS)) assert not missing, f"ledger rows deleted (ids are permanent): {missing}" - assert len(EXPECTED_INITIAL_IDS) == 71 + assert len(EXPECTED_INITIAL_IDS) == 75 def test_version_tuple_pads_to_three_components(): @@ -589,6 +598,23 @@ def test_all_membership_helper_semantics(): "removed_in 4.0 is due", (3, 9, 0), ), + # introduced_in cannot be dodged via 'evaluate' (behavior-row bypass regression) + ( + { + "id": "M-907", + "kind": "behavior", + "old": "diff_diff:BaconDecompositionResults", + "new": None, + "introduced_in": "3.9", + "deprecated_in": None, + "removed_in": None, + "status": "evaluate", + "phase": 2, + }, + (3, 9, 0), + "evaluate cannot satisfy an introduction", + (3, 8, 0), + ), # overdue param-value removal: value migrations are due-gated like symbol rows ( { @@ -626,6 +652,7 @@ def test_all_membership_helper_semantics(): "overdue-introduce-only-alias", "overdue-env-default-decision", "overdue-removal", + "introduced-in-evaluate-bypass", "overdue-param-value-removal", "early-removal-before-schedule", ],