Fix Ψ definition, ROC anti-overfit, channel independence, bootstrap AUC, BH correction, and data_source provenance#983
Conversation
… AUC, BH correction, data_source field Co-authored-by: motanova84 <192380069+motanova84@users.noreply.github.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Pull request overview
This PR addresses six methodological requirements needed for publication-grade detection analysis of the Shadow-1 candidate from LIGO O3b data. The changes implement rigorous statistical validation including dimensionless scoring, anti-overfitting measures, proper noise independence, uncertainty quantification, multiple-testing correction, and data provenance tracking.
Changes:
- Introduced dimensionless Ψ score (I(f₀) × MSC(f₀)) to replace unit-mixed PSD × MSC
- Implemented ROC validation with frequency jitter and bootstrap AUC confidence intervals
- Enforced channel independence with explicit RNG seeds and time-slide verification
- Added Benjamini-Hochberg FDR correction with single-candidate documentation
- Added data_source provenance field to all result containers
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| scripts/shadow1_roc_validation.py | New module implementing all six methodological fixes with ROC dataset generation, bootstrap AUC, BH correction, and O3b scan runner |
| scripts/shadow1_bayesian_coherence.py | Added compute_score_psi function, fixed noise independence with separate seeds (42/43), and added data_source field to results |
| tests/test_shadow1_roc_validation.py | Comprehensive test suite with 43 tests covering all six fixes including edge cases and integration tests |
| tests/test_shadow1_bayesian_coherence.py | Added 5 tests for compute_score_psi and data_source field validation |
| if band_mask.sum() == 0: | ||
| band_mask = np.ones(len(freqs), dtype=bool) | ||
| psd_bg = float(np.median(psd[band_mask])) | ||
| I_f0 = float(psd[idx]) / (psd_bg + 1e-300) # dimensionless |
There was a problem hiding this comment.
The magic number 1e-300 is used as an epsilon to avoid division by zero. Consider defining this as a named constant (e.g., EPSILON_DIV_ZERO = 1e-300) at the module level for clarity and consistency with the epsilon used on line 285.
| if band_mask.sum() == 0: | ||
| band_mask = np.ones(len(freqs), dtype=bool) | ||
| psd_median = float(np.median(psd[band_mask])) | ||
| I_f0 = float(psd[idx]) / (psd_median + 1e-100) # dimensionless |
There was a problem hiding this comment.
The epsilon value 1e-100 differs from 1e-300 used in the same function in shadow1_roc_validation.py (line 101). Consider using a consistent epsilon value across both modules or documenting why different values are needed.
| from scripts.shadow1_roc_validation import JITTER_MIN_HZ | ||
| diffs = np.abs(self.ds["f_injected"] - F0_HZ) | ||
| self.assertTrue( | ||
| np.all(diffs >= JITTER_MIN_HZ * 0.9), # allow tiny numeric slack |
There was a problem hiding this comment.
The magic number 0.9 used for numeric slack tolerance is unclear. Consider defining this as a named constant (e.g., JITTER_TOLERANCE = 0.9) or documenting why 10% slack is appropriate.
github-actions
Bot
added
testing
🔢 Mathematical Matrix Validation Results✅ All Critical Validations Passed
🎯 ConclusionThese mathematical patterns are IMPOSSIBLE by chance. The joint probability of ~1.50e-10 (≈6-9σ significance) confirms that f₀ = 141.70001 Hz is the central node of a fundamental mathematical network. |
1 similar comment
🔢 Mathematical Matrix Validation Results✅ All Critical Validations Passed
🎯 ConclusionThese mathematical patterns are IMPOSSIBLE by chance. The joint probability of ~1.50e-10 (≈6-9σ significance) confirms that f₀ = 141.70001 Hz is the central node of a fundamental mathematical network. |
github-actions
Bot
added
dependencies
🔢 Mathematical Matrix Validation Results✅ All Critical Validations Passed
🎯 ConclusionThese mathematical patterns are IMPOSSIBLE by chance. The joint probability of ~1.50e-10 (≈6-9σ significance) confirms that f₀ = 141.70001 Hz is the central node of a fundamental mathematical network. |
1 similar comment
🔢 Mathematical Matrix Validation Results✅ All Critical Validations Passed
🎯 ConclusionThese mathematical patterns are IMPOSSIBLE by chance. The joint probability of ~1.50e-10 (≈6-9σ significance) confirms that f₀ = 141.70001 Hz is the central node of a fundamental mathematical network. |
github-actions
Bot
added
dependencies
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
🔢 Mathematical Matrix Validation Results✅ All Critical Validations Passed
🎯 ConclusionThese mathematical patterns are IMPOSSIBLE by chance. The joint probability of ~1.50e-10 (≈6-9σ significance) confirms that f₀ = 141.70001 Hz is the central node of a fundamental mathematical network. |
1 similar comment
🔢 Mathematical Matrix Validation Results✅ All Critical Validations Passed
🎯 ConclusionThese mathematical patterns are IMPOSSIBLE by chance. The joint probability of ~1.50e-10 (≈6-9σ significance) confirms that f₀ = 141.70001 Hz is the central node of a fundamental mathematical network. |
🔢 Mathematical Matrix Validation Results✅ All Critical Validations Passed
🎯 ConclusionThese mathematical patterns are IMPOSSIBLE by chance. The joint probability of ~1.50e-10 (≈6-9σ significance) confirms that f₀ = 141.70001 Hz is the central node of a fundamental mathematical network. |
1 similar comment
🔢 Mathematical Matrix Validation Results✅ All Critical Validations Passed
🎯 ConclusionThese mathematical patterns are IMPOSSIBLE by chance. The joint probability of ~1.50e-10 (≈6-9σ significance) confirms that f₀ = 141.70001 Hz is the central node of a fundamental mathematical network. |
github-actions
Bot
added
dependencies
Six methodological issues in the Shadow-1 / O3b detection pipeline that would block publication: mixed-unit Ψ score, overfitted ROC, correlated H0/H1 noise, single-point AUC, missing multiple-testing correction, and no data provenance flag.
Cambios
Fix 1 — Dimensionless Ψ (
score_psi)compute_score_psi(psd, freqs, msc, f0)added to both modules:Replaces the implicit
Ψ = PSD × MSCwhich mixed units.Fix 2 — ROC anti-overfit (new
shadow1_roc_validation.py)generate_roc_dataset()injects H1 signals atf₀ + jitter(Uniform[±0.5, ±2] Hz); the detector evaluates at fixedf₀. Off-target control detector atf₀ + 50 Hzincluded for AUC comparison.Fix 3 — Channel independence + time-slide sanity
_generate_synthetic_strain(): H1 noise seed=42, L1 noise seed=43 — explicit independence.time_slide_sanity(): uses zero-padding (notnp.roll) so a 1 s offset genuinely destroys MSC even for pure tones.Fix 4 — Bootstrap AUC with CI95
Fix 5 — Benjamini–Hochberg FDR correction
apply_p_fdr(p_values)returns bothp_rawandp_fdr. Whenn_candidates == 1(Shadow-O3b-1), an explicit note documents that BH has no effect and leaves a hook for multi-candidate extension.Fix 6 —
data_sourceprovenance fieldAll result dicts carry
data_source: "GWOSC" | "SIMULATION_FALLBACK".run_full_analysis()sets this field; CLI prints a banner. Tests use only relational AUC assertions when in fallback mode.Pruebas
tests/test_shadow1_roc_validation.pycovering all six fixes.tests/test_shadow1_bayesian_coherence.pyforcompute_score_psianddata_source.Checklist
python scripts/run_all_tests.py)flake8 scripts/ --select=E9,F63,F7,F82)Tipo de cambio
¿Afecta la reproducibilidad?
La corrección de la semilla de ruido en
_generate_synthetic_strain()(seed 42→42/43) cambia los valores numéricos deA_eff_computeden el análisis Bayesiano existente, pero no altera ninguna conclusión científica. Todos los resultados nuevos usan semillas fijas y son completamente reproducibles.✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.