perf(efficient_did): hoist kcov tables across cells + per-group W lifecycle (100k: 86->18 min) (#629)

Author: igerber

CHANGELOG.md

@@ -242,6 +242,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   the legacy pseudoinverse path as described under `omega_ridge` above; at `omega_ridge=0`
   results are bit-identical to the previous release. The conditional-path scalability warning
   now fires once per fit at n > 50,000 (previously per cell at n > 5,000).
+- **`EfficientDiD` conditional-path kernel-covariance tables hoisted across `(g,t)` cells**
+  (follow-up to the fused tiled pass above). Every Omega* term is a kernel covariance keyed only
+  by outcome columns, so the tables the tiled pass rebuilt per cell dedup to one table of
+  distinct product columns per comparison group per unit-tile (~26x fewer kernel GEMM columns
+  on a PT-All fit); kernel weight matrices are built and freed one group at a time, so the tile
+  memory budget is governed by the largest single group instead of the sum (proportionally
+  fatter tiles at large n); each cell's Omega* is gathered from the group tables through
+  compact per-cell column slices in the legacy per-entry operation order (value-exact, locked
+  by test). Measured (3-rep medians, 20 periods, 5 cohorts, 5 covariates, `aggregate="all"`):
+  fit 7.7s → 7.0s at 2k units, 57.2s → 35.7s at 10k (1.6x), 358s → 129s at 30k (2.8x), and
+  the 100k-unit fit drops from ~86 min to **17.8 min (~4.8x)** at the same ~4 GB peak RSS;
+  the kernel-covariance stage itself is 40x faster (21.2s → 0.5s at 10k). Results move only at
+  floating-point reassociation level (post-treatment cells ~1e-12 relative, overall ATT ~1e-13);
+  the no-covariates path is byte-identical and `omega_ridge=0` still routes the entire legacy
+  path. A `pt_assumption="post"` covariate fit builds no tables at all (all cells are
+  just-identified), keeping that path regression-free.
 - **CallawaySantAnna multiplier bootstrap rewritten as a fused, column-tiled scatter-GEMM**
   (EfficientDiD's bootstrap routes through the same kernel). The former loop sliced the
   `(block × n_units)` weight matrix twice per (g,t) cell per weight block — at a 40-period,

TODO.md

@@ -46,7 +46,7 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m
 
 | Issue | Location | Origin | Effort | Priority |
 |-------|----------|--------|--------|----------|
-| `EfficientDiD` conditional path at very large n: the fused tiled GEMM rewrite (v3.7; 43x at 2k units, 100k now completes in ~86 min) computes kernel-covariance tables per cell per tile. Cross-cell table hoisting — Term 5 tables are (g,t)-independent and Term 2 tables depend only on t — would cut the intrinsic per-cell GEMM factor by ~10x at 100k-unit scale if practitioners need routine 100k covariate fits. Follow-up lever from the v3.7 flop analysis; not needed below ~50k units. | `efficient_did_covariates.py::compute_conditional_cells_tiled` | CS-scaling | Mid | Low |
+| `EfficientDiD` conditional path: post-hoisting (2026-07, 100k fit ~18 min) the top stage at every scale is `_ridge_solve_weights` — batched `np.linalg.solve` LU over `(n × H × H)` stacks (17.2s of a 35.7s fit at 10k units, ~170s at 100k; linear in n). The matrices are SPD after the ridge, so a batched Cholesky (potrf ~1/2 the LU flops) in the Rust backend — which could also thread across the stack — is the natural lever; numpy has no batched triangular solve, so pure-Python gains are limited. | `efficient_did_covariates.py::_ridge_solve_weights`, `rust/src/` | CS-scaling | Mid | Low |
 | `ImputationDiD` dense `(A0'A0).toarray()` scales `O((U+T+K)^2)` — OOM risk on large panels (only triggers when the sparse solver fails). Needs an alternative dense fallback or richer sparse strategy. | `imputation.py` | #141 | Heavy | Medium |
 | CR2 Bell-McCaffrey DOF uses a naive `O(n²k)` per-coefficient loop over cluster pairs; Pustejovsky-Tipton (2018) Appendix B has a scores-based formulation avoiding the full `n×n` `M`. Switch when a user hits a large-`n` cluster-robust design. | `linalg.py::_compute_cr2_bm` | Phase 1a | Heavy | Low |
 | Rust-backend HC2: the Rust path only supports HC1; HC2 and CR2 Bell-McCaffrey fall through to NumPy. Noticeable for large-`n` fits. | `rust/src/linalg.rs` | Phase 1a | Mid | Low |