Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -120,3 +120,8 @@ tools/reviewer-eval/runs/
benchmarks/refresh_2026_07/venvs/
benchmarks/refresh_2026_07/results/raw/
benchmarks/refresh_2026_07/results/refresh_results_smoke.json

# Stata batch mode writes <basename>.log to the cwd (repo root) - never committed;
# the golden JSON it produces IS committed (benchmarks/data/lpdid_ra_stata_golden.json).
# Root-anchored to the generator's own log so it doesn't hide logs elsewhere.
/generate_lpdid_ra_golden.log
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added
- **LP-DiD regression-adjustment SE now anchored against Stata `teffects` (no library
behavior change).** The RA influence-function cluster SE has no runnable R reference
(`alexCardazzi/lpdid` does direct covariate inclusion, not RA), so it was previously
pinned against itself + MC-coverage-validated. It is now cross-validated against Stata
`teffects ra ... atet vce(cluster)` via a committed golden
(`benchmarks/stata/generate_lpdid_ra_golden.do` → `lpdid_ra_stata_golden.json`;
`tests/test_lpdid_ra_stata_parity.py`): the library matches `teffects` to ~1e-16 at all
7 event-study horizons, confirming the no-finite-sample-factor convention (REGISTRY
LPDiD Deviation 2). This is the repo's first Stata parity arm — `teffects` is native
(no SSC dependency) and goldens are committed, so CI needs no Stata.

### Changed
- **Internal tracking docs reorganized (no library behavior change).** `TODO.md` is now
the actionable backlog only; blocked / parked work and won't-fix decisions moved to the
Expand Down
2 changes: 1 addition & 1 deletion DEFERRED.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ exists but parity can't be verified without a local toolchain.
| `CallawaySantAnna` bootstrap: align p-value computation with R `did`'s symmetric-percentile method (former "CallawaySantAnna Bootstrap Improvements" section). | `staggered.py` | — | Low |
| **`bias_corrected_local_linear` (lprobust) Phase-1c follow-ups:** extend golden parity to `kernel ∈ {triangular, uniform}` (epa-only today); expose `vce ∈ {hc0,hc1,hc2,hc3}` on the public wrapper once R goldens exist (port supports all four; needs a per-mode generator + a hc2/hc3 q-fit-leverage decision); clustered-DGP auto-bandwidth parity is **blocked upstream** on an nprobust singleton-cluster bug in `lpbwselect.mse.dpi` (Phase-1c DGP 4 uses manual `h=b=0.3`). | `_nprobust_port.py`, `local_linear.py`, `generate_nprobust_lprobust_golden.R` | Phase 1c | Low-Med |
| `HeterogeneousAdoptionDiD` Stute-family Stata-bridge parity: no public R `Stutetest` package exists; would add `benchmarks/stata/generate_stute_golden.do` + a Stata dependency. | `benchmarks/stata/`, `tests/test_stute_test_parity.py` | follow-up | Low |
| **`LPDiD` regression-adjustment SE — no runnable R reference.** The RA influence-function cluster SE is canonically Stata `teffects ra ... atet vce(cluster)` only; no R package computes it (`alexCardazzi/lpdid` does direct covariate inclusion, not RA). Today the RA *point* is R-anchored (~1e-12), the SE is pinned + MC-coverage-validated (`coverage_lpdid_ra.py`). Follow-up: contribute the RA path to `alexCardazzi/lpdid` so a runnable R RA reference exists — only a *trusted* anchor once cross-checked vs Stata `teffects` (else circular). | `tests/test_methodology_lpdid.py`, `benchmarks/python/coverage_lpdid_ra.py` | #B2 follow-up | Low |
| **`LPDiD` regression-adjustment SE — Stata-`teffects` anchored; optional R follow-up only.** The RA influence-function cluster SE is canonically Stata `teffects ra ... atet vce(cluster)` (no R package computes it — `alexCardazzi/lpdid` does direct covariate inclusion, not RA). **Resolved:** `benchmarks/stata/generate_lpdid_ra_golden.do` reconstructs each horizon's clean sample independently and anchors the SE against `teffects` to ~1e-16 at all 7 horizons (golden `lpdid_ra_stata_golden.json`; `tests/test_lpdid_ra_stata_parity.py`, which also gates the Stata row count vs the library `n_obs`); the point is triple-anchored (Python/R/Stata). Stata is node-locked (not in CI), so the CI-runnable guards remain the regression pin (`RA_SE_PIN`) + MC coverage (`coverage_lpdid_ra.py`). *Optional residual (no longer a blocker):* contribute an RA path to `alexCardazzi/lpdid` so a runnable **R** reference also exists. | `benchmarks/stata/generate_lpdid_ra_golden.do`, `tests/test_lpdid_ra_stata_parity.py` | #B2 · Stata arm | Low |
| **`LPDiD` survey scope gaps (PR-D1 deferrals).** Survey support covers the variance-weighted default path only. (a) `survey_design` + `reweight=True` (the equally-weighted / regression-adjustment IF path) is rejected: the weighted RA influence-function variance has **no runnable survey reference** (same class as the RA-SE row above - `survey::svyglm` anchors only the OLS/WLS path). (b) Replicate-weight survey designs (BRR/Fay/JK1/JKn/SDR) and (c) non-pweight (fweight/aweight) types are rejected pending demand. | `lpdid.py`, REGISTRY #8 | PR-D1 | Low |
| **`LPDiD` non-absorbing SE/RA external-reference gap.** The non-absorbing *point* estimands are R-parity-locked (PR-C2: independent `fixest::feols` Eq. 12/13 reconstruction, point+SE ~1e-13/~1e-15 vw; `effect_stabilization` reweighted point + pinned SE), but the authors' canonical non-absorbing SE/RA is Stata `lpdid`/`teffects` only - no faithful R analogue exists (`alexCardazzi/lpdid`'s `nonabsorbing_lag` proved NOT a faithful Eq. 13: off-switch clamp + non-paper boundary/placebo window, diverges ~0.01-0.05 even on a monotone panel; recorded as a divergent reference, not a gate). Revisit if a Stata toolchain or a corrected R package appears - same class as the absorbing RA-SE row above. | `benchmarks/R/generate_lpdid_golden.R`, `tests/test_methodology_lpdid.py` | PR-C2 | Low |
| `HeterogeneousAdoptionDiD` Phase-3 R-parity: ships coverage-rate validation on synthetic DGPs, not tight point parity vs `chaisemartin::stute_test` / `yatchew_test` (needs bootstrap-seed-semantics + `B` alignment across numpy/R). | `tests/test_had_pretests.py` | Phase 3 | Low |
Expand Down
21 changes: 20 additions & 1 deletion benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,21 @@ Required R packages:
- `jsonlite` - JSON interchange
- `data.table` - Fast data manipulation

### Stata

A few goldens are anchored against Stata where no runnable R reference exists.
Stata is node-locked single-user, so — like the R arm — the goldens are committed
and CI never needs Stata. Generators live in `stata/` and are run headless:

```bash
# macOS, StataSE 19 (binary is not on PATH by default)
/Applications/Stata/StataSE.app/Contents/MacOS/stata-se -b do \
benchmarks/stata/generate_lpdid_ra_golden.do
```

Current arm uses only **native** Stata commands (`teffects`) — no SSC packages —
so `version 19` pins behavior. See `stata/README.md`.

## Directory Structure

```
Expand All @@ -56,6 +71,9 @@ benchmarks/
│ ├── benchmark_synthdid.R # Synthetic DiD
│ ├── benchmark_honest.R # HonestDiD
│ └── benchmark_fixest.R # Basic DiD / TWFE
├── stata/
│ ├── README.md # Stata arm docs
│ └── generate_lpdid_ra_golden.do # LPDiD RA SE vs teffects ra
├── python/
│ ├── utils.py # Common utilities
│ ├── benchmark_callaway.py # CallawaySantAnna
Expand All @@ -72,11 +90,12 @@ benchmarks/

## Estimator Comparisons

| diff-diff | R Package | Reference | Status |
| diff-diff | Reference Package | Reference | Status |
|-----------|-----------|-----------|--------|
| `CallawaySantAnna` | `did::att_gt` | Callaway & Sant'Anna (2021) | ✓ Integrated |
| `SyntheticDiD` | `synthdid::synthdid_estimate` | Arkhangelsky et al. (2021) | ✓ Integrated |
| `DifferenceInDifferences` | `fixest::feols` | Standard DiD | ✓ Integrated |
| `LPDiD` (RA SE) | Stata `teffects ra ... atet` | Dube, Girardi, Jorda & Taylor (2025) | ✓ Integrated |
| `HonestDiD` | `HonestDiD::createSensitivityResults` | Rambachan & Roth (2023) | Planned |

Note: HonestDiD benchmark scripts exist but are not yet integrated into the main runner.
Expand Down
24 changes: 24 additions & 0 deletions benchmarks/data/lpdid_ra_stata_golden.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"meta": {
"estimator": "LPDiD regression-adjustment (RA) SE - Stata teffects ra atet",
"generator": "benchmarks/stata/generate_lpdid_ra_golden.do",
"source_panel": "benchmarks/data/lpdid_test_panel.csv",
"point_anchor": "benchmarks/data/lpdid_golden.json ra_cov[h][0]",
"stata_edition": "SE",
"cmd": "teffects ra (Dy x i.time) (tdiff), atet vce(cluster unit)",
"se_convention": "cluster-robust IF at unit, NO finite-sample factor (teffects convention); t(G-1) reference on the library side",
"note": "Independent Stata reconstruction of the per-horizon clean sample (ports generate_lpdid_golden.R prep/clean_h). N and G are emitted to power the Python row-count sample-integrity gate.",
"stata_version": 19.0,
"pre_window": 3,
"post_window": 4
},
"ra_se": {
"0": {"att": 2.888871838355342, "se": 0.45516589747218866, "N": 522, "G": 60},
"1": {"att": 3.225268910679945, "se": 0.44522904812910252, "N": 461, "G": 60},
"2": {"att": 3.5779502216280248, "se": 0.46179396866595729, "N": 401, "G": 60},
"3": {"att": 3.973278128236954, "se": 0.39295354484873846, "N": 353, "G": 60},
"4": {"att": 4.2651037814454096, "se": 0.39398602601018534, "N": 305, "G": 60},
"-2": {"att": 0.1204444745695894, "se": 0.43592937799131937, "N": 461, "G": 60},
"-3": {"att": -0.11970460945194859, "se": 0.43296260860283592, "N": 401, "G": 60}
}
}
133 changes: 133 additions & 0 deletions benchmarks/stata/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Stata parity benchmarks

Stata golden generators live here. They are the **first Stata arm** in the repo;
the pattern mirrors `benchmarks/R/` (a `generate_*` script writes a committed
golden JSON that a skip-guarded `tests/test_*_parity.py` reads, so CI never needs
Stata). Stata is node-locked single-user, so — exactly like the R arm — goldens
are committed and only regenerated locally.

Locating the binary (macOS, StataSE 19; **not** on `PATH` by default):

```
/Applications/Stata/StataSE.app/Contents/MacOS/stata-se
```

---

# `teffects ra` parity for the LP-DiD regression-adjustment SE

`benchmarks/stata/generate_lpdid_ra_golden.do` produces
`benchmarks/data/lpdid_ra_stata_golden.json`, consumed by
`tests/test_lpdid_ra_stata_parity.py` to verify that diff-diff's LP-DiD
regression-adjustment (RA) standard error (Dube, Girardi, Jorda & Taylor 2025)
matches Stata `teffects ra ... atet vce(cluster)` across the 7 event-study
horizons.

## Why Stata `teffects`

The RA covariate path reports an influence-function cluster variance with **no
finite-sample factor**. **No R package computes it** — `alexCardazzi/lpdid` uses
direct covariate inclusion, not RA — so there is no runnable R analogue. The
canonical reference is Stata `teffects ra ... atet vce(cluster)`, which the LP-DiD
authors themselves invoke (paper footnote 9). `teffects` is a **native** Stata
command (no SSC package), so `version 19` in the generator fully pins its
numerical behavior; there is no third-party package version to track.

This arm converts REGISTRY `## LPDiD` Deviation 2 from an *inference* (the
no-finite-sample-factor convention was argued from degrees-of-freedom comments in
the authors' `.do` files) into a *measurement*: the library RA IF SE matches
`teffects` to ~1e-16 at every horizon (same machine / BLAS).

## What the generator does

It reads (does **not** regenerate) `benchmarks/data/lpdid_test_panel.csv` — whose
sole owner is `benchmarks/R/generate_lpdid_golden.R` — and **independently
reconstructs** each horizon's clean sample by porting the R `prep` + `clean_h`
recipe (`generate_lpdid_golden.R:97-123`) to `tsset` + `L.`/`F.` time-series
operators. This gives three independent sample constructions (Python / R / Stata);
a port bug is very likely to be caught before it could masquerade as an SE finding,
because the Python test's point gate (Stata ATET vs the R-anchored `ra_cov[h][0]`) and
sample-shape gate (`(e(N), e(N_clust))` vs the library's `(n_obs, n_clusters)`) would
fail first.

`tsfill` is load-bearing: it mirrors R `fill_gaps()` so the absorbing `treat`
recompute and the long differences see the completed calendar grid across the
deliberate interior gap at `(unit==60, time==7)`.

The per-horizon command is:

```stata
teffects ra (Dy x i.time) (tdiff), atet vce(cluster unit)
```

(The authors' footnote-9 syntax omits the covariate; the RA *covariate* path adds
`x` to the outcome model, as here.)

## Regenerating

```bash
# from the repo root
/Applications/Stata/StataSE.app/Contents/MacOS/stata-se -b do \
benchmarks/stata/generate_lpdid_ra_golden.do

# batch mode ALWAYS exits 0 — verify the log has no Stata errors:
grep -E '^r\([0-9]+\);' generate_lpdid_ra_golden.log # must print nothing
```

The output JSON is **committed** so CI doesn't need Stata. Only re-run when the
committed panel changes (then also refresh `RA_SE_PIN` in
`tests/test_methodology_lpdid.py`) or the horizon set changes.

## Skip behavior

`tests/test_lpdid_ra_stata_parity.py` calls `pytest.skip(...)` when the golden (or
the shared panel / R golden it cross-checks against) is absent, so CI passes
without Stata. The estimator itself is exercised independently by
`tests/test_methodology_lpdid.py` and `tests/test_lpdid.py`.

## JSON schema

```json
{
"meta": {
"estimator": "LPDiD regression-adjustment (RA) SE - Stata teffects ra atet",
"generator": "benchmarks/stata/generate_lpdid_ra_golden.do",
"source_panel": "benchmarks/data/lpdid_test_panel.csv",
"point_anchor": "benchmarks/data/lpdid_golden.json ra_cov[h][0]",
"stata_edition": "SE",
"cmd": "teffects ra (Dy x i.time) (tdiff), atet vce(cluster unit)",
"se_convention": "...",
"note": "...",
"stata_version": 19.0,
"pre_window": 3,
"post_window": 4
},
"ra_se": {
"0": {"att": <float>, "se": <float>, "N": <int>, "G": <int>},
"...": "post {0..4} then pre {-2,-3}; h=-1 is the omitted reference"
}
}
```

Values are written at `%21.17g` (round-trip-exact IEEE-754 double). `N` and `G`
(`e(N)` / `e(N_clust)`) are emitted so the Python gate can assert the Stata clean
sample matches the library's realized size and cluster count — which, together with
the point (1e-10) and SE (~1e-16) agreement, strongly corroborates the same clean
sample. No timestamp — the golden regenerates byte-identically.

## Known constraints

- **Batch mode always exits 0**, even on a hard error (`r(NNN);`). Never trust the
shell exit code — parse the `.log` for `^r\([0-9]+\);` (the generator also runs
an in-`.do` point smoke gate at 1e-8 that surfaces as `r(9);` on a gross port
bug; the Python test's 1e-10 gate is authoritative).
- **`c(flavor)` misreports the edition** as `IC` on StataSE, and `c(edition)` is
unreliable. The generator derives the edition from the `c(MP)` / `c(SE)` 0/1 flags,
with BE by elimination (`c(BE)` is undefined and `cond()` evaluates all branches
eagerly, so it must not be referenced). `"SE"` is simply the committed golden's
current value.
- **SSC has no version history.** `ssc install` always fetches latest and there is
no lockfile / archive to pin against. This arm is exempt because `teffects` is
native (pinned by `version 19`), but any *future* SSC-dependent Stata generator
must record `which <pkg>` output verbatim into its golden's `meta` so drift is
at least detectable.
Loading
Loading