Skip to content

Commit ac7c191

Browse files
igerberclaude
andcommitted
feat: MMM calibration export (diff_diff.mmm) - PyMC-Marketing lift tests + Meridian ROI priors
Interop builders that assemble Marketing Mix Model calibration inputs from experiment results. Explicit-in / validated-out: the caller supplies the already-scoped incremental outcome and its SE (aggregated to the population and window one MMM row represents), and diff-diff assembles the target schema, enforces each consumer's guards, converts to the lognormal parameterization, and pools. It does NOT rescale a headline ATT - that reconciliation needs the MMM's row granularity, time window, and outcome scale, which the exporter cannot see. Pure numpy/pandas; imports no MMM package; introspects no result object, so the module is purely additive (touches no estimator). - to_pymc_marketing_lift_test(channel, x, delta_x, delta_y, sigma, dims=, on_wrong_sign=): builds the lift-test DataFrame consumed by pymc-marketing's MMM.add_lift_test_measurements (prophetverse-compatible). Guards: sigma>0, delta_x!=0, x>=0, x+delta_x>=0, finite delta_y; wrong-sign (NonMonotonicError) AND zero-lift (degenerate for the Gamma lift likelihood) share one on_wrong_sign policy; dims reserved-column + shared-keys validation. - to_meridian_roi_prior(incremental_outcome, incremental_outcome_se, spend, parameter="roi_m"|"mroi_m", se_widening=): builds Google Meridian lognormal priors (MeridianROIPrior); mu/sigma match meridian's lognormal_dist_from_mean_std (verified 1.7.0), spend-weighted multi-experiment pooling, non-positive-ROI and finiteness guards, and a channel- and time-scoped .to_code() snippet that sets media_prior_type. roi_m vs mroi_m selects the estimand and each parameter's Meridian default for non-experiment channels. - 43 behavioral tests (schema, all guards, sign/zero policy, lognormal roundtrip via scipy, pooling, to_code scoping, realistic DiD/CS workflow) + end-to-end smoke verified against real pymc-marketing 0.19.4 (frame accepted into the model graph; wrong-signed row raises NonMonotonicError upstream). - Docs: REGISTRY interop section, api/mmm.rst, llms.txt + llms-full.txt, README line, references.rst, doc-deps.yaml, CHANGELOG; TODO/DEFERRED follow-up rows (Meridian mask builder, PR-B tutorial, post-4.0 result-derived scaling via results.aggregate()). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014wAAYSVo1Yf3XMcxNMrqsj
1 parent 12cccb2 commit ac7c191

14 files changed

Lines changed: 1372 additions & 1 deletion

File tree

CHANGELOG.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
6868
standard rename rows (M-088/M-089) under the spec's missed-rename clause.
6969
Docs + ledger + test-count updates only - **no public API or numerical behavior
7070
change.**
71+
- **MMM calibration export (`diff_diff.mmm`)** - interop builders that assemble
72+
Marketing Mix Model calibration inputs from experiment results, with no MMM
73+
package dependency and no result introspection (purely additive). Design is
74+
explicit-in / validated-out: the caller supplies the already-scoped
75+
incremental outcome and its SE (aggregated to the population and window one
76+
MMM row represents), and diff-diff assembles the schema, enforces each
77+
consumer's guards, and converts to the target parameterization - it does not
78+
rescale a headline ATT, because that reconciliation needs the MMM's row
79+
structure and outcome scale it cannot see.
80+
`to_pymc_marketing_lift_test(channel, x, delta_x, delta_y, sigma, dims=,
81+
on_wrong_sign=)` builds the PyMC-Marketing/prophetverse lift-test DataFrame
82+
with sign/zero (Gamma-likelihood)/positivity guards.
83+
`to_meridian_roi_prior(incremental_outcome, incremental_outcome_se, spend,
84+
parameter="roi_m"|"mroi_m", se_widening=)` builds Google Meridian lognormal
85+
ROI/mROI priors (`MeridianROIPrior` with `mu`/`sigma` matching Meridian's
86+
`lognormal_dist_from_mean_std`, spend-weighted multi-experiment pooling, and a
87+
channel- and time-scoped `.to_code()` snippet that sets `media_prior_type`).
88+
Deriving totals directly from a fitted result is deferred to the post-4.0
89+
`results.aggregate()` layer.
90+
7191
- **Reviewer-eval harness: N-arm matrix, blinded grading, corpus grown 2 -> 11
7292
(`tools/reviewer-eval/`, prep for the GPT-5.6 reviewer evaluation).**
7393
`config/configs.json` moves from the two-arm `control`/`candidate` shape to an

DEFERRED.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ For survey-specific limitations (`NotImplementedError` paths), see the
7070

7171
| Issue | Location | PR | Priority |
7272
|-------|----------|----|----------|
73+
| MMM interop: result-derived scaling for the exporters - let a fitted result produce the scoped incremental outcome + SE directly (via the post-4.0 `results.aggregate()` layer, where the estimator owns its aggregation weights/balance/survey masses), so callers need not hand-scale the ATT. v1 is explicit-in by design; this is the seamless follow-up. | `diff_diff/mmm.py`, `docs/v4-design.md` | mmm-interop | Low |
7374
| `SyntheticControl` fit-snapshot residency (`_SyntheticControlFitSnapshot`) — **investigated 2026-07-07, parked**: the snapshot ALIASES the fit's own working pivots (zero extra construction cost); the retained residency implements the documented freeze contract (post-fit mutation of estimator inputs must not change `in_space_placebo()` / `leave_one_out()` / conformal output on an already-returned results object, and `__getstate__` already excludes it from pickles). A compact array representation saves only pandas overhead (the float panel dominates); releasing residency needs new API surface (`release`/opt-out flag) or a freeze-contract change. Revisit on user demand for very large donor panels. | `synthetic_control.py`, `synthetic_control_results.py` | follow-up | Low |
7475
| Stratified survey-PSU multiplier-weight draw-tiling — **investigated 2026-07-07, parked**: the stratified generator (`generate_survey_multiplier_weights_batch`) consumes ONE sequential rng stream stratum-major (`rng.choice(size=(n_bootstrap, n_h))` per stratum, then lonely-PSU pooling), so draw-chunked assembly CANNOT reproduce the stream bit-identically (contra the old row's parenthetical) — it would need per-stratum generator state skipping (PCG64.advance + per-weight-type variate accounting; fragile) or a stream-layout change (MC-level SE changes → baseline/golden recapture + REGISTRY note). Stratified designs have few PSUs, so the full `(n_bootstrap × n_psu)` matrix rarely matters; unstratified (the large-`n_units` case) is already tiled. Revisit only if a large-PSU stratified design hits memory, as a documented stream change. | `diff_diff/bootstrap_chunking.py::iter_survey_multiplier_weight_blocks` | follow-up | Low |
7576
| ChangesInChanges FULL Melly-Santangelo covariate estimator (monotonized integrated-indicator conditional CDFs, treated-post `F_{X|11}` integration, exchangeable bootstrap with variance-weighted KS bands, tail trimming, pre-period specification test). The qte-`xformla` simplified form of the MS pipeline SHIPPED in the covariates PR (`covariates=` on both estimators, parity-tested vs qte 1.3.1); this row's earlier "No R parity target exists" claim was WRONG and is corrected in the MS review doc. The full estimator has no reference implementation (the MS Stata code is the only one; distinct from Kranker's `cic`) and would need simulation-based validation. Reviewed: `docs/methodology/papers/melly-santangelo-2015-review.md`. | `diff_diff/changes_in_changes.py` | #682 | Low |

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ Full guide: `diff_diff.get_llm_guide("practitioner")`.
130130
- [Honest DiD](https://diff-diff.readthedocs.io/en/stable/api/honest_did.html) - Rambachan & Roth (2023) sensitivity analysis: robust CI under PT violations, breakdown values
131131
- [Pre-Trends Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/pretrends.html) - Roth (2022) minimum detectable violation and power curves
132132
- [Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/power.html) - analytical and simulation-based MDE, sample size, power curves for study design
133+
- [MMM Calibration Export](https://diff-diff.readthedocs.io/en/stable/api/mmm.html) - convert experiment results into MMM calibration inputs: PyMC-Marketing lift-test frames and Google Meridian lognormal ROI priors
133134
- Conley spatial HAC SE (`vcov_type="conley"`) on cross-sectional `LinearRegression` / `compute_robust_vcov` plus panel `DifferenceInDifferences` / `MultiPeriodDiD` / `TwoWayFixedEffects` (with `conley_lag_cutoff` for within-unit Bartlett temporal HAC) - Conley (1999) spatial-correlation-aware SEs with parity vs R `conleyreg` on cross-sectional + panel fixtures, optional combined spatial + cluster product kernel via explicit `cluster=`, auto-activating sparse k-d-tree fast path for `n > 5_000`
134135

135136
## Survey Support

TODO.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,5 +43,7 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m
4343
|-------|----------|--------|--------|----------|
4444
| Align the four legacy dataset loaders (`load_card_krueger`, `load_castle_doctrine`, `load_divorce_laws`, `load_mpdta`) with the loud-fallback pattern of `load_prop99`/`load_walmart`: `UserWarning` + `df.attrs["source"]` marker on synthetic fallback (currently silent), plus optional checksum pinning for the CSV downloads. **Upgraded to a live defect 2026-07-13: the `causaldata/causal_datasets` GitHub repo backing castle/card_krueger/divorce is dead (404), so those loaders silently serve synthetic data everywhere - needs loud fallback + replacement sources.** | `diff_diff/datasets.py` | LWDiD precursor | Quick | Medium |
4545
| Tighten the mypy suppressions that back the enforced-zero posture: burn down `prep_dgp`'s per-module `[index]` override (needs a None-vs-array restructure that preserves the seeded RNG stream), and evaluate re-enabling the globally disabled codes (`arg-type`, `return-value`, `var-annotated`, `assignment`) one at a time — `assignment` alone hid several real annotation drifts found during the 2026-07 triage. | `pyproject.toml` `[tool.mypy]`, `diff_diff/prep_dgp.py` | lint-CI | Mid | Low |
46+
| MMM interop follow-up: Meridian `roi_calibration_period` mask builder - accept the MMM's time index + channel order and emit the boolean `(n_media_times, n_media_channels)` mask so `.to_code()` scopes the prior to the experiment window automatically (today the caller passes a mask expression / `full_model_window=True`). | `diff_diff/mmm.py` | mmm-interop | Quick | Low |
47+
| MMM interop PR-B: calibration tutorial notebook (fit DiD/CS -> scope -> `to_pymc_marketing_lift_test` / `to_meridian_roi_prior`) + a `llms-practitioner.txt` Step 8 pointer to the exporters as the MMM hand-off. | `docs/tutorials/`, `diff_diff/guides/llms-practitioner.txt` | mmm-interop | Mid | Low |
4648
| Tracking-file contract guard test: reject NEW active deferred-work pointers at `TODO.md` (deferred rows live in `DEFERRED.md`; allowlist for historical/past-tense prose and actionable-row pointers) and assert rows cross-linking a `docs/v4-deprecations.yaml` `M-xxx` id don't restate ledger status. Origin: tracking-split local review R2. | `tests/`, `TODO.md`, `DEFERRED.md` | tracking-split | Quick | Low |
4749
| Real-data CI canary for dataset-backed replication tests: `test_methodology_lwdid.py`'s Prop 99 / Walmart goldens skip (visibly) when loaders fall back to synthetic; add a lane or canary asserting `df.attrs["source"] == "lwdid_ssc_ancillary"` in CI so network regressions cannot silently de-gate the replication tests. Pairs with the loader-fallback repair row above. | `tests/test_methodology_lwdid.py`, `.github/workflows/` | LWDiD validation suite | Quick | Low |

diff_diff/__init__.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,11 @@
154154
)
155155
from diff_diff.lpdid import LPDiD
156156
from diff_diff.lpdid_results import LPDiDResults
157+
from diff_diff.mmm import (
158+
MeridianROIPrior,
159+
to_meridian_roi_prior,
160+
to_pymc_marketing_lift_test,
161+
)
157162
from diff_diff.power import (
158163
PowerAnalysis,
159164
PowerResults,
@@ -566,6 +571,10 @@
566571
"Alert",
567572
"OutcomeShape",
568573
"TreatmentDoseShape",
574+
# MMM calibration export (interop)
575+
"to_pymc_marketing_lift_test",
576+
"to_meridian_roi_prior",
577+
"MeridianROIPrior",
569578
# LLM guide accessor
570579
"get_llm_guide",
571580
]

diff_diff/guides/llms-full.txt

Lines changed: 79 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2042,7 +2042,6 @@ sample_result = compute_sample_size(effect_size=0.5, sigma=1.0)
20422042

20432043
# Simulation-based power
20442044
sim_result = simulate_power(
2045-
n_units=200, n_periods=8, treatment_period=4,
20462045
effect_sizes=[0.1, 0.5, 1.0, 2.0],
20472046
n_simulations=500, seed=42,
20482047
)
@@ -2731,3 +2730,82 @@ cross-period aggregations are enumerated in
27312730
current release; see that document for phrasing rules, the
27322731
no-traffic-light decision, unit-translation policy, and schema
27332732
stability policy.
2733+
2734+
## MMM Calibration Export
2735+
2736+
Interop builders converting experiment results into Marketing Mix Model
2737+
(MMM) calibration inputs. Pure numpy/pandas - no MMM package is imported,
2738+
no result object is introspected (the module is purely additive). Design:
2739+
EXPLICIT IN, VALIDATED OUT. Reconciling an experiment estimate to a
2740+
calibration input needs the target MMM's row granularity (per-geo vs
2741+
national), its time window, and the outcome scale (additive levels vs
2742+
log/rate/share) - none of which diff-diff can see - so the CALLER supplies
2743+
the already-scoped incremental outcome and its SE (read off summary(),
2744+
aggregated to the population/window one MMM row represents). diff-diff
2745+
assembles the schema, enforces each consumer's guards, converts to the
2746+
lognormal parameterization, pools, and emits snippets. Deriving totals
2747+
from a fit is deferred to the post-4.0 results.aggregate() layer, where
2748+
the estimator owns its aggregation weights/balance/survey masses.
2749+
2750+
```python
2751+
from diff_diff import (
2752+
SyntheticDiD,
2753+
to_pymc_marketing_lift_test,
2754+
to_meridian_roi_prior,
2755+
)
2756+
2757+
result = SyntheticDiD().fit(panel, outcome="revenue", treatment="treated",
2758+
unit="geo", time="week")
2759+
2760+
# PyMC-Marketing / prophetverse lift-test frame. Pass the scoped effect:
2761+
# here the SDID ATT is already the per-treated-geo per-week lift.
2762+
df_lift = to_pymc_marketing_lift_test(
2763+
channel="tv", # must match the MMM's channel_columns
2764+
x=50_000.0, # baseline spend for this row's scope
2765+
delta_x=20_000.0, # spend change (nonzero; negative go-dark,
2766+
# with x + delta_x >= 0)
2767+
delta_y=result.att, # measured lift, scoped to this row
2768+
sigma=result.se, # its SE (finite, > 0)
2769+
dims={"geo": "US-CA"}, # optional model-dim coordinate
2770+
on_wrong_sign="raise", # raise|drop|keep - PyMC rejects
2771+
# sign(delta_y)!=sign(delta_x) AND delta_y==0
2772+
# (degenerate for its Gamma lift likelihood)
2773+
)
2774+
2775+
# Google Meridian lognormal prior. Caller aggregates the ATT to a total
2776+
# incremental outcome over the treated population/window and its SE.
2777+
prior = to_meridian_roi_prior(
2778+
incremental_outcome=180_000.0, # total incremental revenue
2779+
incremental_outcome_se=45_000.0, # its SE
2780+
spend=200_000.0, # channel spend the outcome is over
2781+
parameter="roi_m", # roi_m (full-spend/zero-spend return)
2782+
# | mroi_m (marginal return)
2783+
se_widening=1.5, # >=1 for transferability skepticism
2784+
)
2785+
prior.roi_mean, prior.roi_sd # pooled ROI moments
2786+
prior.mu, prior.sigma # LogNormal params (match Google's
2787+
# lognormal_dist_from_mean_std)
2788+
prior.to_dict() # JSON-ready
2789+
print(prior.to_code( # ready-to-paste PriorDistribution +
2790+
channel="tv", # ModelSpec; roi_m/mroi_m is per-channel
2791+
media_channels=["search", "tv"], # so channel scope is required (vector
2792+
# prior in model channel order; other
2793+
# channels keep the Meridian default),
2794+
roi_calibration_period="mask", # AND time scope (mask expr) or
2795+
# full_model_window=True; sets
2796+
)) # media_prior_type accordingly.
2797+
```
2798+
2799+
Key points:
2800+
- Lift frame: one row per experiment, columns [channel, *dims, x, delta_x,
2801+
delta_y, sigma]; guards on sigma>0, delta_x!=0, x>=0, x+delta_x>=0,
2802+
finite delta_y; dims must not collide with reserved columns and must
2803+
share one key set. on_wrong_sign governs wrong-sign AND zero-lift rows
2804+
(drop-all raises; keep warns the frame is not valid PyMC input).
2805+
- Meridian: roi = incremental_outcome/spend, spend-weighted pooling,
2806+
roi_sd = sqrt(sum((w_i*sd_i)^2)) (independence assumed - widen via
2807+
se_widening); non-positive pooled ROI raises (lognormal positivity);
2808+
all scaled/pooled outputs re-validated finite-positive.
2809+
- parameter="roi_m" vs "mroi_m" picks the Meridian estimand + its default
2810+
for non-experiment channels (LogNormal(0.2,0.9) vs LogNormal(0.0,0.5))
2811+
and the emitted media_prior_type ("roi" vs "mroi").

diff_diff/guides/llms.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ Full practitioner guide: call `diff_diff.get_llm_guide("practitioner")`
8383
- [Honest DiD](https://diff-diff.readthedocs.io/en/stable/api/honest_did.html): Rambachan & Roth (2023) sensitivity analysis — robust CI under parallel trends violations, breakdown values
8484
- [Pre-Trends Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/pretrends.html): Roth (2022) Section II.A-B no-individually-significant (NIS) box-probability pretest power + minimum detectable violation; `pretest_form='nis'` (default) implements the paper's primary form, `pretest_form='wald'` retained as paper-supported alternative (Propositions 1+3+4 all apply); linear-violation MDV in Roth's γ units when relative-time labels are threaded through `fit()`; full Σ_22 routing on non-bootstrap CallawaySantAnna and SunAbraham adapters
8585
- [Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/power.html): Analytical and simulation-based power analysis — MDE, sample size, power curves for study design
86+
- [MMM Calibration Export](https://diff-diff.readthedocs.io/en/stable/api/mmm.html): Assemble Marketing Mix Model calibration inputs from experiment results (explicit-in / validated-out - the caller passes the already-scoped incremental outcome + SE, the module does NOT rescale a headline ATT). `to_pymc_marketing_lift_test(channel, x, delta_x, delta_y, sigma, dims=, on_wrong_sign=)` builds the PyMC-Marketing/prophetverse lift-test DataFrame with sign/zero/positivity guards. `to_meridian_roi_prior(incremental_outcome, incremental_outcome_se, spend, parameter="roi_m"|"mroi_m", se_widening=)` builds Google Meridian lognormal ROI priors (spend-weighted pooling, lognormal parity with `lognormal_dist_from_mean_std`, channel- and time-scoped `.to_code()` snippet setting `media_prior_type`). Pure numpy/pandas; imports no MMM package; does not introspect result objects. Deriving totals from a fit is deferred to the post-4.0 `results.aggregate()` layer.
8687
- Conley spatial HAC SE (`vcov_type="conley"`) on cross-sectional `LinearRegression` / `compute_robust_vcov` PLUS panel `DifferenceInDifferences` / `MultiPeriodDiD` / `TwoWayFixedEffects` (with `conley_lag_cutoff=<int>` for within-unit Bartlett temporal HAC) — Conley (1999) spatial-correlation-aware SEs with haversine/euclidean/callable distance metric and Bartlett/uniform spatial kernel; panel path uses the R `conleyreg`-form block-decomposed sandwich (within-period spatial + within-unit Bartlett serial, same-time excluded); parity vs R `conleyreg` (Düsterhöft 2021) on cross-sectional AND panel `lag_cutoff > 0` fixtures. Combining with explicit `cluster=<col>` applies the combined spatial + cluster product kernel `K_total[i,j] = K_space · 1{c_i = c_j}` (cluster must be constant within each unit across periods on the panel path; validator-enforced). DiD takes `unit=<col>` as a fit-time kwarg when `vcov_type="conley"` (not on `__init__`). Sparse k-d-tree fast path auto-activates for `n > 5_000` with bartlett kernel + haversine/euclidean metric
8788

8889
## Tutorials

0 commit comments

Comments
 (0)