Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/golden-validation-recipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ A correct finding carries the material to **sanity-check** it, not just a verdic
| `verdict` | `supported` | in-window peak from a reliable sample |
| `latency_ms` | `100.0` | the planted peak |
| `expected_window_ms` | `[80.0, 120.0]` | the auditory M100 window |
| `search_window_ms` | in-window bounds | provenance of the measurement (not the old ±100 ms widening) |
| `search_window_ms` | `[30.0, 235.0]` | provenance of the measurement. Floor excludes the ~15 ms artifact; ceiling clears the 215 ms refutation threshold so the claim could actually have failed |
| `n_epochs` | `40` | above `MIN_RELIABLE_EPOCHS`, so the average is trustworthy |
| `channel` | the peak channel | which sensor carried it |
| `caveats` | `[]` | nothing to flag |
Expand All @@ -57,7 +57,7 @@ thin, out-of-window sample:

```python
from matilde_plugin.engine.meg_study import build_golden_steps, MIN_RELIABLE_EPOCHS
steps = build_golden_steps(peak_latency_ms=300.0, n_epochs=MIN_RELIABLE_EPOCHS - 1)
steps = build_golden_steps(peak_latency_ms=225.0, n_epochs=MIN_RELIABLE_EPOCHS - 1)
```

This does **not** return a confident `refuted`. Because the epoch count is below
Expand Down
91 changes: 87 additions & 4 deletions docs/meg-validation-study.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,93 @@ response to standard tones and validates it falls within the expected window.
3. `epoch` — epoch around standard-tone events (the analysis must read the events
from the data, not assume counts); checkpoint epoch metadata.
4. `evoked` — average to the evoked response; checkpoint the evoked array artifact.
5. `validate_finding` — measure peak latency in the M100 window; emit a finding:
`supported` if within the expected window (default 80–120 ms), `refuted` if
clearly outside, `inconclusive` if SNR too low / sample too small. Record the
measured latency + amplitude as evidence.
5. `validate_finding` — measure peak latency in the M100 window; emit a finding.
`supported` if the peak is inside the expected window (default 80–120 ms).
`refuted` only if it is later than the **refutation threshold** — the later of
215 ms (the end of P200 territory) or `window_hi + 60 ms`. Everything else is
`inconclusive`, with a caveat naming *why*. Record latency + amplitude as
evidence.

The peak search runs from **30 ms** (artifact-exclusion floor) to **20 ms past
the refutation threshold**, capped at 280 ms and clamped to the epoch.

### Why the verdict geometry looks like this

The peak is a polarity-blind argmax (`mode="abs"`). Two failure modes pull in
opposite directions, and both have actually happened here:

- **Search too narrow → `refuted` unreachable.** Between `e079993` and
2026-07-30 the search was window ±10 ms while refutation required window
±50 ms, so **no measurement at any configured window could return `refuted`**.
The prediction could not fail.
- **Search too wide → the argmax catches something that is not the M100.** The
original bug was exactly this: a ±100 ms search grabbed the ~15 ms stimulus
artifact. The first attempt to fix the falsifiability problem (2026-07-30)
widened the search to a 40 ms floor while leaving an early refute threshold at
60 ms — reproducing the same incident at a new latency, with a *confident*
`refuted` where the previous code had safely said `inconclusive`.

Width alone cannot resolve that. So the search is wide enough to contain the
refutation threshold, and the classifier refuses to refute anywhere a known
component could be the winner:

```
30 ms 70 ms window 170 ms 215 ms 280 ms
---|-------------|---------[====]-------|---------|------------|---
floor P50/Pa supported off-window P200 refutable -->
(artifact) ambiguous late ambiguous
-> inconclusive -> inconclusive
```

**Early refutation is not achievable this way, and the code does not pretend
otherwise.** Everything early enough to contradict the M100 is also P50/Pa
territory (this dataset has a P50 around 50 ms). Establishing a genuinely absent
M100 needs amplitude/SNR or topography, not latency — see "Still open".

Three guards make the remaining verdicts honest:

- **`boundary_hit`** — an argmax pinned within 1 ms of a search edge means the
true extremum is probably *outside* the range. Forced to `inconclusive`.
- **`component_ambiguous`** — a peak in P50 or P200 territory is named as such
rather than scored.
- **`unfalsifiable_late`** — if the clamped epoch cannot reach the refutation
threshold, `supported` was the only attainable answer, and it is labelled
untested rather than confirmed.

> **Falsifiability invariant, test-enforced:** the search range must strictly
> contain the late refutation threshold. `_SEARCH_BEYOND_THRESHOLD_MS` is the
> margin that guarantees it. If you narrow the search to chase an artifact, raise
> the floor — do not let the floor cross a refute threshold. Enforced by
> `test_late_refutation_is_reachable_from_the_real_search_range`,
> `test_search_range_contains_the_late_refutation_threshold`,
> `test_peak_at_the_search_floor_is_not_a_confident_refutation` and
> `test_falsifiability_holds_under_the_shipped_epoch_bounds`; the constants are
> mutation-tested (7/7 mutants killed).

**Consequences for the record.** The single historical `refuted` predates the
artifact fix and came from measuring the 15 ms artifact — **retracted**; it was
never evidence against the M100. Every `supported` recorded between `e079993`
and 2026-07-30 carried **no falsification risk**. Re-running the *golden* recipe
does not re-test anything (its synthetic backend returns a planted constant); the
number that needs re-measuring is the real **120.0 ms** from `e079993`, which sat
exactly on the old search's upper bound and disagrees with this dataset's
published N100 (~95 ms) by 25 ms.

### Still open

- **Absence cannot be refuted.** `get_peak` always returns an argmax, so "no
M100" is indistinguishable from "some peak somewhere". Needs an amplitude/SNR
threshold.
- **Early refutation** needs polarity or topography, per above.
- **The band magnitudes (70 / 170 / 215 ms) are conventional, not fitted.** They
bracket the canonical N1 (60–150 ms) and P2 (150–250 ms) and the M100's
reported ~99–153 ms range across frequency and intensity (Roberts et al. 2000),
but they are not derived from this dataset. Refutation therefore requires a
large latency departure — it is a guard against gross misidentification, not a
sensitive test of latency shifts.
- **`MIN_RELIABLE_EPOCHS = 8`** gates `refuted` but never `supported`. That
asymmetric evidentiary bar deserves its own review; 8 is also well below CAEP
practice.

## Implementation rules (match the repo)

Expand Down
Loading
Loading