fix(v4): ETWFE reference-period normalization (#724) + Stata jwdid/csdid parity

CHANGELOG.md

@@ -7,7 +7,112 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-### Added
+### Fixed
+- **`WooldridgeDiD` silently dropped genuine post-treatment effects
+  (issue #724).** With `control_group="never_treated"`, the ETWFE design emitted
+  every cohort×time indicator *including* the reference period — which is
+  collinear with the absorbed unit fixed effects — and left generic QR rank
+  detection to remove whichever column it happened to pick. On the real
+  Callaway-Sant'Anna `mpdta` panel that removed `ATT(2004, 2004)` and
+  `ATT(2006, 2007)`, returning **5 of 7** post-treatment cells with only a
+  rank-deficiency warning to explain it.
+  The reference is not a free choice: Wooldridge (2025) Eq. 6.1 excludes
+  `dg_i·f(g-1)_t` "so that `s = g − 1` is the reference period", and Stata
+  `jwdid ... never` omits the same cell. The design now omits
+  `ref(g) = max{t : t < g − anticipation AND cohort g is observed at t}`
+  explicitly — per-cohort, since a panel-wide rule omits an all-zero column
+  whenever a cohort is unobserved there and leaves the collinearity intact.
+  All 7 cells now return with **no rank warning**, matching `jwdid`'s cell set
+  exactly and its ATTs to ~1e-14.
+  Two related cases are handled rather than left to QR: `(g, t)` pairs a cohort
+  never observes are skipped (and named in a warning, gated on
+  `rank_deficient_action`), and a cohort with no pre-treatment period at all is
+  **excluded from the estimation sample** with a warning naming it — dropping
+  only its columns would leave its rows in the omitted baseline, loading its
+  treatment effect onto the time fixed effects and biasing the cohorts that are
+  identified. Where that exclusion removes the last comparison group, `fit()`
+  now raises instead of returning an all-NaN result.
+  The bug was **not** confined to `never_treated`: any cohort whose cell block
+  spans its own cohort dummy hits it, which `anticipation > 0` makes reachable
+  on the default `not_yet_treated` path.
+- **`WooldridgeDiD` no longer returns a NaN overall ATT as a successful fit.**
+  Several designs previously produced a result object with
+  `overall_att = overall_se = NaN` and no error, which reads as a completed
+  estimate: every treatment cell removed by rank reduction, every cell falling
+  inside the anticipation window (`overall_att` averages `t >= g`, while cells
+  from `t >= g − anticipation` are estimated), and post-treatment coefficients
+  all rank-dropped. `fit()` now raises with a cause-specific message. Verified
+  pre-existing rather than introduced by the reference-period work. The
+  invariant covers the **point estimate only** — `overall_se` and inference may
+  still be NaN where the variance alone is unidentifiable (`hc2_bm` without
+  Bell-McCaffrey DOF, `aggregate(weights="cohort_share")`), both fail-closed by
+  design. Note a design with **two** surviving treated cohorts can still be
+  degenerate when neither post cell has a same-period comparison, so cohort
+  count is not a proxy for comparison support.
+- **`rank_deficient_action="error"` is honored for unobserved cohort×time
+  cells.** Those cells are now skipped before the solve rather than left to QR.
+  The first version of that change treated `"error"` and `"warn"` identically,
+  which silently downgraded an explicit fail-closed request to a warning; all
+  three modes now dispatch, and `"error"` raises naming the skipped pairs.
+
+### Changed
+- **`WooldridgeDiD` now raises when rank reduction removes any requested
+  cohort×time cell**, instead of reporting the survivors under their original
+  `ATT(g,t)` labels. Once a cell is dropped, the remaining coefficients
+  identify whatever contrast the reduced design supports — typically a
+  difference between two treated cohorts — so the labels no longer describe the
+  estimand. Measured on an all-eventually-treated panel with a true ATT of 1.5
+  in every post cell: `ATT(3,8) = -0.0096`, `ATT(5,9) = -0.2393`, and
+  `overall_att = 1.0023`, with **no warning** under
+  `rank_deficient_action="silent"`.
+  **This is a capability regression, taken deliberately.** Panels with no
+  never-treated group — where periods in which every unit is treated have no
+  eligible comparison — previously produced those wrong numbers and now fail
+  closed, including under `survey_design=` and `cohort_trends=True`. Wooldridge
+  (2025) Section 5.4 defines the correct treatment (the last cohort serves as
+  the control, its columns dropped deliberately rather than by QR); the library
+  already applies that to the cohort-trend column but not to the cells, and
+  implementing the cell half is tracked to restore these fits.
+- **`WooldridgeDiD` refuses `survey_design=` combined with an unidentified
+  cohort** (`NotImplementedError`, all three methods). Excluding a cohort under
+  a complex survey design is domain estimation, which this path does not
+  implement: deleting the rows also removes their PSUs and strata from the
+  Taylor-linearized variance and from `df_survey = n_PSU − n_strata`, so
+  surviving ATTs would carry variance computed on a design the user never
+  specified (measured on a two-stratum panel: `df_survey` 22 → 14, finite SE, no
+  diagnostic). The correct behavior is zero-padding per REGISTRY *Subpopulation
+  Analysis* / Lumley (2004) §3.4; until that lands the combination is refused
+  rather than approximated. Restrict the frame to identified cohorts yourself,
+  reduce `anticipation`, or fit without `survey_design=`. Tracked in `TODO.md`.
+- **Survey designs are validated before any row is dropped.** The full
+  `SurveyDesign` (weights, `weight_type`, strata, PSU, FPC, nesting) is now
+  resolved against the pre-exclusion sample. Previously validation ran after
+  cohort exclusion, so invalid metadata confined to an excluded cohort — NaN /
+  negative / infinite weights, fractional `fweight`, missing PSU or strata —
+  vanished with those rows and the fit returned finite estimates instead of
+  rejecting the design.
+
+### Added
+- **Stata parity arm for ETWFE and Callaway-Sant'Anna ATT(g,t) (`jwdid` / `csdid`).**
+  `benchmarks/stata/generate_etwfe_cs_golden.do` anchors both staggered
+  estimators against their canonical Stata implementations on the genuine
+  `mpdta` panel (committed at its pinned SHA-256, so the arm never needs
+  network). Both match their reference to ~3e-8 on every post-treatment cell.
+  This **replaces a test that was validating nothing**: `test_wooldridge.py`
+  asserted ETWFE ATT(g,t) *equals* CS ATT(g,t) within `5e-3`, which is false on
+  real data — at `(2007, 2007)` they differ by `0.0171`. It passed only because
+  `load_mpdta()` was silently substituting a synthetic, effect-homogeneous DGP
+  when its source 404'd (issue #722). Stata's `jwdid` and `csdid` reproduce the
+  same disagreement, establishing it as a property of the estimators rather
+  than a defect in either.
+  The arm also **surfaced a real deviation**: `WooldridgeDiD`'s default `hc1`
+  SEs are uniformly smaller than `jwdid`'s, making them anti-conservative
+  (~0.1% at G=500, ~2.8% at G=20 — the gap grows as clusters shrink).
+  `CallawaySantAnna` shows no such gap. The mechanism is **not yet
+  identified** — it tracks `sqrt(G/(G-1))` but sits above it, and `solve_ols`
+  already applies the full CR1 — so the deviation is recorded and pinned as
+  measured rather than diagnosed; see the REGISTRY `## WooldridgeDiD (ETWFE)`
+  note and the `TODO.md` row.
 - **Post-fit aggregation: `results.aggregate(type=...)` on CallawaySantAnna
   (4.0 program Phase 2b, ledger rows M-020 / M-117 / M-122).** Estimate once,
   aggregate as a post-fit step — the ecosystem norm (`did::aggte`,

DEFERRED.md

@@ -46,7 +46,7 @@ exists but parity can't be verified without a local toolchain.
 | `StaggeredTripleDifference` R cross-validation: CSV fixtures not committed (gitignored); tests skip without local R + `triplediff`. Commit fixtures or generate deterministically. | `tests/test_methodology_staggered_triple_diff.py` | #245 | Medium |
 | `StaggeredTripleDifference` R parity: benchmark only tests the no-covariate path (`xformla=~1`). Add covariate-adjusted scenarios + aggregation-SE parity assertions. | `benchmarks/R/benchmark_staggered_triplediff.R` | #245 | Medium |
 | `StaggeredTripleDifference` per-cohort group-effect SEs include WIF (conservative vs R's `wif=NULL`); documented in REGISTRY. Could override the mixin for an exact R match (verification needs R `triplediff`). | `staggered_triple_diff.py` | #245 | Low |
-| **WooldridgeDiD follow-up cluster** (PR-B Stage D/E fail-closed surfaces; re-enable after R/Stata validation): <br>• QMLE sandwich uses `aweight` cluster adjustment `(G/(G-1))·(n-1)/(n-k)` vs Stata's `G/(G-1)` (conservative); add a `qmle` weight type if Stata goldens confirm a material difference (`wooldridge.py`, `linalg.py`). <br>• response-scale APE / log-link coefficient bridge for R `etwfe(family=poisson|logit)` cell-level parity — needs `emfx()` APE extraction or link-inversion with baseline-mean adjustment (`generate_wooldridge_golden.R`, `test_methodology_wooldridge.py`). <br>• `aggregate(weights="cohort_share")` on survey-weighted fits: `_n_g_per_cohort` uses raw `unit.nunique()`; implement design-weighted unit totals per cohort (paper W2025 §7) and lift the `ValueError` gate (`wooldridge.py`, `wooldridge_results.py`). <br>• unconditional inference for `cohort_share` accounting for ω̂_g sampling uncertainty (W2025 §7.5); currently NaN-closed (`wooldridge_results.py`). <br>• `cohort_trends=True × survey_design` and `× control_group="never_treated"` raise `NotImplementedError` (unvalidated TSL variance / trend columns spanned by placebo cell-dummies) (`wooldridge.py`). <br>• Stata `jwdid` golden-value `TestReferenceValues` (`tests/test_wooldridge.py`). | `wooldridge.py`, `wooldridge_results.py`, `linalg.py`, benchmarks | #216 · PR-B | Med-Low |
+| **WooldridgeDiD follow-up cluster** (PR-B Stage D/E fail-closed surfaces; re-enable after R/Stata validation): <br>• QMLE sandwich uses `aweight` cluster adjustment `(G/(G-1))·(n-1)/(n-k)` vs Stata's `G/(G-1)` (conservative); add a `qmle` weight type if Stata goldens confirm a material difference (`wooldridge.py`, `linalg.py`). <br>• response-scale APE / log-link coefficient bridge for R `etwfe(family=poisson|logit)` cell-level parity — needs `emfx()` APE extraction or link-inversion with baseline-mean adjustment (`generate_wooldridge_golden.R`, `test_methodology_wooldridge.py`). <br>• `aggregate(weights="cohort_share")` on survey-weighted fits: `_n_g_per_cohort` uses raw `unit.nunique()`; implement design-weighted unit totals per cohort (paper W2025 §7) and lift the `ValueError` gate (`wooldridge.py`, `wooldridge_results.py`). <br>• unconditional inference for `cohort_share` accounting for ω̂_g sampling uncertainty (W2025 §7.5); currently NaN-closed (`wooldridge_results.py`). <br>• `cohort_trends=True × survey_design` and `× control_group="never_treated"` raise `NotImplementedError` (unvalidated TSL variance / trend columns spanned jointly by the placebo cells and the unit FE, which absorb the cohort indicator and recover the omitted reference) (`wooldridge.py`). <br>• Stata `jwdid` golden-value `TestReferenceValues` (`tests/test_wooldridge.py`). | `wooldridge.py`, `wooldridge_results.py`, `linalg.py`, benchmarks | #216 · PR-B | Med-Low |
 | Extend `WooldridgeDiD` `method ∈ {logit, poisson}` with `vcov_type ∈ {classical, hc2, hc2_bm}`: composing HC2 leverage + Bell-McCaffrey DOF with the QMLE pseudo-residual sandwich needs derivation + R parity vs `clubSandwich::vcovCR(glm, type="CR2")`. Rejected at `__init__`. | `wooldridge.py` | follow-up | Medium |
 | Multi-constraint CR2 parallel-trends test (AHT/HTZ) for `hc2_bm` fits: DiagnosticReport's PT check routes `vcov_type="hc2_bm"` sources to Bonferroni over the BM-adjusted per-row p-values because the generic chi-square joint Wald would discard the CR2 small-sample correction (see REPORTING.md "hc2_bm parallel-trends policy"). The proper joint test is the AHT/HTZ Wald with a Satterthwaite-style denominator df over the pre-period contrast block; needs derivation for the stacked/pooled WLS-CR2 layout + parity vs `clubSandwich::Wald_test(..., test="HTZ")`. | `diagnostic_report.py`, `linalg.py` | vcov/df round-trip PR | Low |
 | `PreTrendsPower` CS/SA `anticipation=1` R-parity fixture: R `pretrends` has no anticipation parameter, so the Python `_extract_pre_period_params` anticipation filter isn't R-parity-locked. Build a synthetic CS/SA result with `anticipation=1` and assert γ_p matches R's `slope_for_power()`. (Mechanism already covered by MC + full-VCV tests.) | `tests/test_methodology_pretrends.py`, `generate_pretrends_golden.R` | PR-C | Low |