From 47c1f90d6da982d134f10bd28c1867fbc94ad9fe Mon Sep 17 00:00:00 2001 From: Guillain d'Erceville <167749917+Guillain-RDCDE@users.noreply.github.com> Date: Sun, 14 Jun 2026 17:47:13 +0200 Subject: [PATCH] feat(scoring): gate Rule 1's near-Nyquist 320 kbps signature on wall hardness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A 320 kbps MP3 low-passes at ~20.5 kHz, where genuinely band-limited lossless (baroque, harpsichord, older mastering, world-music reissues) also rolls off. Rule 1 flagged both as a "320 kbps spectral" transcode from cutoff position alone — on a full-library audit that was ~65% of all FAKE_CERTAIN verdicts, most of them authentic. Rule 1 now measures the residual spectral floor above the wall: a real 320k brickwall drops to digital silence (<= -55 dB vs the in-band reference), an authentic rolloff keeps a higher analog/dither floor (> -55 dB). Above the threshold the signature is dropped; at/below it the file stays FAKE_CERTAIN. A single threshold rather than a 3-band scheme on purpose: an intermediate band that withholds the +50 lets the score fall below the fast FAKE_CERTAIN short-circuit, after which protective rules (e.g. R7 clean-silence -50) can wrongly clear a genuine transcode — observed on a confirmed real @320 file. Calibrated on 50 synthetic FLAC->320k pairs plus a band-limited surrogate (residual ROC AUC 0.95) and verified end-to-end. Residual is computed only for near-Nyquist cutoffs (90-95% Nyquist) to keep the hot path fast; unknown/short inputs fall back to the previous behaviour, so only that zone changes. --- CHANGELOG.md | 17 ++++ src/flac_detective/analysis/analyzer.py | 3 +- .../analysis/new_scoring/calculator.py | 4 + .../analysis/new_scoring/models.py | 3 + .../analysis/new_scoring/rules/spectral.py | 45 +++++++++- .../analysis/new_scoring/strategies.py | 1 + src/flac_detective/analysis/spectrum.py | 86 +++++++++++++++++-- tests/test_new_scoring_rules.py | 69 +++++++++++++++ 8 files changed, 220 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ccfa080..7fef0c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,20 @@ +## Unreleased + +- **Fewer false positives on band-limited music (Rule 1 near-Nyquist gate).** A 320 kbps + MP3 low-passes at ~20.5 kHz — exactly where genuinely band-limited lossless (baroque, + harpsichord, 1960s–80s mastering, world-music reissues) also rolls off. Rule 1 used to + flag both as a "320 kbps spectral" transcode from the cutoff position alone, which on a + full-library audit accounted for ~65% of all FAKE_CERTAIN verdicts — most of them + authentic. Rule 1 now measures the **residual spectral floor above the wall**: a real + 320k brickwall drops to digital silence, while an authentic rolloff keeps an + analog/dither floor. Above −55 dB the signature is dropped (→ AUTHENTIC); at or below it + the file stays FAKE_CERTAIN. Calibrated on 50 synthetic FLAC→320k pairs plus a + band-limited surrogate (ROC AUC 0.95) and verified against a confirmed real transcode. + **This changes verdicts**: near-Nyquist files previously marked FAKE_CERTAIN on + band-limited material now read AUTHENTIC. Only the near-Nyquist 320 kbps zone at + 44.1 kHz is affected; all other detection paths are unchanged, and unknown/short inputs + fall back to the previous behaviour. + ## v1.4.0 (2026-06-10) — Beets plugin + English-only output An adoption-focused feature release: FLAC Detective now plugs into beets, speaks diff --git a/src/flac_detective/analysis/analyzer.py b/src/flac_detective/analysis/analyzer.py index b472610..84b6319 100644 --- a/src/flac_detective/analysis/analyzer.py +++ b/src/flac_detective/analysis/analyzer.py @@ -104,7 +104,7 @@ def analyze_file(self, filepath: Union[str, Path]) -> Dict: duration_check = check_duration_consistency(temp_path, metadata) # Spectral analysis (OPTIMIZED: uses cache -> points to TEMP) - cutoff_freq, energy_ratio, cutoff_std = analyze_spectrum( + cutoff_freq, energy_ratio, cutoff_std, residual_floor_db = analyze_spectrum( temp_path, self.sample_duration, cache=cache ) @@ -128,6 +128,7 @@ def analyze_file(self, filepath: Union[str, Path]) -> Dict: cache=cache, source_path=filepath, deep=self.deep, + residual_floor_db=residual_floor_db, ) # Add note if analysis was partial diff --git a/src/flac_detective/analysis/new_scoring/calculator.py b/src/flac_detective/analysis/new_scoring/calculator.py index fbec5ac..5562ed4 100644 --- a/src/flac_detective/analysis/new_scoring/calculator.py +++ b/src/flac_detective/analysis/new_scoring/calculator.py @@ -317,6 +317,7 @@ def new_calculate_score( cache=None, source_path: Optional[Path] = None, deep: bool = False, + residual_floor_db: float = float("nan"), ) -> Tuple[int, str, str, str]: """Calculate score using the new 8-rule system with file caching. @@ -332,6 +333,8 @@ def new_calculate_score( analysed audio is a decoded WAV (ALAC/APE). See _calculate_bitrate_metrics. deep: Run Rule 12 on every file, bypassing the authentic fast path (slower; catches silent-heuristic AAC/Vorbis transcodes). See the ``--deep`` flag. + residual_floor_db: Spectral floor above the ~20.5 kHz wall (NaN = unknown). + Drives Rule 1's near-Nyquist 320 kbps wall-hardness gate. """ logger.debug("OPTIMIZATION: File read cache ENABLED (via AudioCache)") @@ -373,6 +376,7 @@ def new_calculate_score( cutoff_freq=cutoff_freq, cutoff_std=cutoff_std, energy_ratio=energy_ratio, + residual_floor_db=residual_floor_db, cache=cache, # Pass shared cache to context ) diff --git a/src/flac_detective/analysis/new_scoring/models.py b/src/flac_detective/analysis/new_scoring/models.py index c11c703..3a7e623 100644 --- a/src/flac_detective/analysis/new_scoring/models.py +++ b/src/flac_detective/analysis/new_scoring/models.py @@ -37,6 +37,9 @@ class ScoringContext: cutoff_freq: float cutoff_std: float = 0.0 energy_ratio: float = 0.0 + # Residual spectral floor above the ~20.5 kHz wall (NaN = unknown / not in the + # near-Nyquist 320 kbps zone). Drives Rule 1's wall-hardness gate. + residual_floor_db: float = float("nan") # State updated during scoring mp3_bitrate_detected: Optional[int] = None diff --git a/src/flac_detective/analysis/new_scoring/rules/spectral.py b/src/flac_detective/analysis/new_scoring/rules/spectral.py index 293686e..81247a5 100644 --- a/src/flac_detective/analysis/new_scoring/rules/spectral.py +++ b/src/flac_detective/analysis/new_scoring/rules/spectral.py @@ -1,19 +1,38 @@ """Spectral analysis rules (Rule 1, Rule 2, Rule 8).""" import logging +import math from typing import List, Optional, Tuple from ..bitrate import estimate_mp3_bitrate, get_cutoff_threshold logger = logging.getLogger(__name__) - -def apply_rule_1_mp3_bitrate( +# Wall-hardness gate for the near-Nyquist 320 kbps zone (Rule 1). +# A 320 kbps cutoff (~20.5 kHz) sits where an authentic band-limited rolloff and a +# real MP3 brickwall overlap, so cutoff position alone cannot tell them apart — this +# is the dominant false-positive source in real libraries. The residual spectral +# floor above the wall does separate them (calibrated on 50 synthetic FLAC->320k +# pairs + a band-limited surrogate, ROC AUC 0.95; the one confirmed real near-Nyquist +# transcode floored at -57 dB): +# residual > -55 dB -> authentic analog/dither floor -> drop the signature +# residual <= -55 dB -> digital-silence floor (transcode) -> keep the +50 (FAKE) +# A single threshold (not a 3-band scheme): an intermediate "gray" band that withholds +# the +50 lets the score fall below the fast FAKE_CERTAIN short-circuit, after which +# protective rules (e.g. R7 clean-silence -50) can wrongly clear a genuine transcode. +# Keeping real transcodes at +50 preserves that short-circuit. Threshold favours the +# project's "protect authentic first" rule, accepting that transcodes of already +# band-limited material (shallow wall, floor > -55 dB) are near-undetectable anyway. +NEARNYQ_FLOOR_DB = -55.0 + + +def apply_rule_1_mp3_bitrate( # noqa: C901 cutoff_freq: float, container_bitrate: float, cutoff_std: float = 0.0, sample_rate: int = 44100, energy_ratio: float = 0.0, + residual_floor_db: float = float("nan"), ) -> Tuple[Tuple[int, List[str]], Optional[int]]: """Apply Rule 1: Constant MP3 Bitrate Detection (Spectral Estimation). @@ -30,6 +49,9 @@ def apply_rule_1_mp3_bitrate( cutoff_std: Standard deviation of cutoff frequency sample_rate: Sample rate in Hz (default: 44100) energy_ratio: High frequency energy ratio (default: 0.0) + residual_floor_db: Spectral floor above the ~20.5 kHz wall in dB (NaN = + unknown). Gates the near-Nyquist 320 kbps branch; NaN falls back to the + legacy cutoff-only behaviour. Returns: Tuple of ((score_delta, list_of_reasons), estimated_bitrate) @@ -126,6 +148,25 @@ def apply_rule_1_mp3_bitrate( # Le bitrate conteneur est-il dans la plage attendue ? if min_br <= container_bitrate <= max_br: + # Near-Nyquist 320 kbps wall-hardness gate. The cutoff alone cannot tell a + # 320k brickwall from an authentic band-limited rolloff here; the residual + # floor can. NaN (unknown / not in the near-Nyquist zone) -> legacy +50. + if ( + estimated_bitrate == 320 + and not math.isnan(residual_floor_db) + and residual_floor_db > NEARNYQ_FLOOR_DB + ): + logger.info( + f"RULE 1: 320 kbps signature dropped (residual floor " + f"{residual_floor_db:.0f} dB > {NEARNYQ_FLOOR_DB:.0f} → authentic band-limited)" + ) + reasons.append( + f"R1: 320 kbps cutoff near Nyquist but high residual floor " + f"({residual_floor_db:.0f} dB) → authentic band-limited, signature dropped" + ) + return (score, reasons), None + # residual <= -55 dB (or unknown): digital-silence floor = real transcode. + score += 50 reasons.append(f"Constant MP3 bitrate detected (Spectral): {estimated_bitrate} kbps") logger.info( diff --git a/src/flac_detective/analysis/new_scoring/strategies.py b/src/flac_detective/analysis/new_scoring/strategies.py index 968e684..2d1b76e 100644 --- a/src/flac_detective/analysis/new_scoring/strategies.py +++ b/src/flac_detective/analysis/new_scoring/strategies.py @@ -50,6 +50,7 @@ def apply(self, context: ScoringContext) -> None: context.cutoff_std, context.audio_meta.sample_rate, context.energy_ratio, + residual_floor_db=context.residual_floor_db, ) context.add_score(score, reasons) context.mp3_bitrate_detected = estimated_bitrate diff --git a/src/flac_detective/analysis/spectrum.py b/src/flac_detective/analysis/spectrum.py index da319c6..55d0612 100644 --- a/src/flac_detective/analysis/spectrum.py +++ b/src/flac_detective/analysis/spectrum.py @@ -17,9 +17,73 @@ logger = logging.getLogger(__name__) +def _welch_magnitude_db( + data_mono: np.ndarray, samplerate: int, nfft: int = 16384 +) -> Tuple[Optional[np.ndarray], Optional[np.ndarray]]: + """Welch-averaged magnitude spectrum (Hann, 50% overlap) in dB. + + A stable spectrum estimate for the residual-floor metric. Returns (None, None) + if the signal is shorter than one FFT window. + """ + if len(data_mono) < nfft: + return None, None + win = get_hann_window(nfft) + step = nfft // 2 + acc: Optional[np.ndarray] = None + count = 0 + with set_workers(1): + for start in range(0, len(data_mono) - nfft, step): + seg = data_mono[start : start + nfft] * win + m = np.abs(rfft(seg)) + acc = m if acc is None else acc + m + count += 1 + if count == 0 or acc is None: + return None, None + mag = acc / count + freq = rfftfreq(nfft, 1 / samplerate) + magnitude_db = 20 * np.log10(mag + 1e-10) + return freq, magnitude_db + + +def compute_residual_floor_db( + full_audio: np.ndarray, samplerate: int, max_seconds: float = 30.0 +) -> float: + """Residual spectral floor just above the ~20.5 kHz wall, vs the in-band reference. + + A real 320 kbps MP3 brickwall drops to the digital-silence floor (strongly + negative, ~ -70 dB); an authentic band-limited rolloff keeps a higher + analog/dither floor (~ -25 to -50 dB). This is the discriminator that cutoff + position alone cannot provide near Nyquist (calibrated on 50 synthetic + FLAC->320k pairs + a band-limited surrogate; see Rule 1's near-Nyquist gate). + + Returns NaN when the reference/top bands are unavailable (e.g. hi-res) or the + signal is too short — callers must treat NaN as "unknown" and fall back to the + legacy behaviour. + """ + try: + data = full_audio + if data.ndim > 1: + data = np.mean(data, axis=1) + data = np.asarray(data[: int(max_seconds * samplerate)], dtype=np.float64) + freq, magnitude_db = _welch_magnitude_db(data, samplerate) + if freq is None or magnitude_db is None: + return float("nan") + nyq = samplerate / 2.0 + ref_mask = (freq >= 0.45 * nyq) & (freq <= 0.65 * nyq) + top_mask = (freq >= 0.961 * nyq) & (freq <= 0.993 * nyq) + if not (np.any(ref_mask) and np.any(top_mask)): + return float("nan") + ref = float(np.median(magnitude_db[ref_mask])) + top = float(np.median(magnitude_db[top_mask])) + return top - ref + except Exception as e: # pragma: no cover - defensive + logger.debug(f"Residual floor computation failed: {e}") + return float("nan") + + def analyze_spectrum( filepath: Path, sample_duration: float = 30.0, cache: "Optional[AudioCache]" = None -) -> Tuple[float, float, float]: +) -> Tuple[float, float, float, float]: """Analyzes the frequency spectrum of the audio file. Takes multiple samples at different times for robustness. @@ -31,10 +95,12 @@ def analyze_spectrum( cache: Optional AudioCache instance for optimization. Returns: - Tuple (cutoff_frequency, energy_ratio, cutoff_std) where: + Tuple (cutoff_frequency, energy_ratio, cutoff_std, residual_floor_db) where: - cutoff_frequency: detected cutoff frequency in Hz - energy_ratio: energy ratio in high frequencies - cutoff_std: standard deviation of cutoff frequency + - residual_floor_db: floor above the ~20.5 kHz wall (NaN unless the cutoff + sits in the near-Nyquist 320 kbps zone, where Rule 1 needs it) """ try: # Create cache if not provided @@ -129,16 +195,26 @@ def _analyze_sample(i: int) -> Tuple[float, float]: # Authentic FLACs often have high variance in cutoff frequency cutoff_std = float(np.std(cutoff_freqs)) if len(cutoff_freqs) > 1 else 0.0 + # Residual-floor metric for Rule 1's near-Nyquist 320 kbps gate. Only the + # ~90-95% Nyquist band needs it (where a 320k brickwall overlaps an authentic + # rolloff), so we skip the extra Welch pass everywhere else to keep the hot + # path fast. + nyquist = samplerate / 2.0 + residual_floor_db = float("nan") + if 0.90 * nyquist <= final_cutoff < 0.95 * nyquist: + residual_floor_db = compute_residual_floor_db(full_audio, samplerate) + logger.info( f"Spectrum analysis: cutoff={final_cutoff:.0f} Hz, " - f"energy_ratio={final_energy:.6f}, cutoff_std={cutoff_std:.1f}, samples={cutoff_freqs}" + f"energy_ratio={final_energy:.6f}, cutoff_std={cutoff_std:.1f}, " + f"residual_floor_db={residual_floor_db:.1f}, samples={cutoff_freqs}" ) - return final_cutoff, final_energy, cutoff_std + return final_cutoff, final_energy, cutoff_std, residual_floor_db except Exception as e: logger.debug(f"Spectral analysis error: {e}") - return 0, 0, 0 + return 0, 0, 0, float("nan") def detect_cutoff( # noqa: C901 diff --git a/tests/test_new_scoring_rules.py b/tests/test_new_scoring_rules.py index 71bb394..fd9c5cb 100644 --- a/tests/test_new_scoring_rules.py +++ b/tests/test_new_scoring_rules.py @@ -1,3 +1,4 @@ +from flac_detective.analysis.new_scoring import estimate_mp3_bitrate from flac_detective.analysis.new_scoring.rules import ( apply_rule_1_mp3_bitrate, apply_rule_2_cutoff, @@ -313,3 +314,71 @@ def test_5_authentic_low_quality(self): assert total_score == 0 verdict, _ = determine_verdict(total_score) assert verdict == "AUTHENTIC" + + +class TestRule1NearNyquistWallGate: + """Rule 1's wall-hardness gate for the near-Nyquist 320 kbps zone. + + A 320 kbps cutoff (~20.5 kHz) overlaps an authentic band-limited rolloff, so the + verdict is gated on the residual spectral floor (the dominant real-world + false-positive source) instead of cutoff position alone. Single threshold at + -55 dB: above = authentic (drop the signature), at/below = real transcode (keep + the +50). A NaN residual falls back to the legacy behaviour. + """ + + SR = 44100 + CUTOFF = 20250 # in the 320 zone, below the 94% Nyquist skip (20727 Hz) + CONTAINER = 850 # within the 320 kbps range (700-1050) + + def test_cutoff_maps_to_320(self): + """Guard: the chosen cutoff must actually be estimated as 320 kbps.""" + assert estimate_mp3_bitrate(self.CUTOFF) == 320 + + def test_clear_high_floor_drops_signature(self): + """High residual floor (> -55 dB, authentic analog) -> signature dropped, no R3.""" + (score, reasons), bitrate = apply_rule_1_mp3_bitrate( + self.CUTOFF, self.CONTAINER, 0.0, self.SR, residual_floor_db=-49.0 + ) + assert score == 0 + assert bitrate is None # R3 will not fire -> authentic + assert any("authentic band-limited" in r for r in reasons) + + def test_keep_real_transcode_floor(self): + """Floor just below -55 dB (real near-Nyquist transcode) keeps the +50. + + Mirrors the one confirmed real ``@320`` transcode (floor -57 dB): it must stay + FAKE so the fast short-circuit fires before protective rules can clear it. + """ + (score, reasons), bitrate = apply_rule_1_mp3_bitrate( + self.CUTOFF, self.CONTAINER, 0.0, self.SR, residual_floor_db=-57.0 + ) + assert score == 50 + assert bitrate == 320 + assert any("Constant MP3 bitrate detected" in r for r in reasons) + # R1 + R3 reach FAKE_CERTAIN + score_r3, _ = apply_rule_3_source_vs_container(bitrate, self.CONTAINER) + verdict, _ = determine_verdict(score + score_r3) + assert verdict == "FAKE_CERTAIN" + + def test_keep_deep_digital_floor_scores_fake(self): + """Floor deep below -55 dB (digital-silence) -> real transcode, +50.""" + (score, _), bitrate = apply_rule_1_mp3_bitrate( + self.CUTOFF, self.CONTAINER, 0.0, self.SR, residual_floor_db=-75.0 + ) + assert score == 50 + assert bitrate == 320 + + def test_unknown_residual_falls_back_to_legacy(self): + """Unknown residual (NaN: non-44.1k / short file) keeps the legacy +50 behaviour.""" + (score, _), bitrate = apply_rule_1_mp3_bitrate(self.CUTOFF, self.CONTAINER, 0.0, self.SR) + assert score == 50 + assert bitrate == 320 + + def test_gate_only_applies_to_320(self): + """A lower-bitrate signature with a residual value is unaffected by the gate.""" + # 17458 Hz -> 192 kbps; container 700 within (500, 750) + (score, _), bitrate = apply_rule_1_mp3_bitrate( + 17458, 700, 0.0, self.SR, residual_floor_db=-30.0 + ) + assert score == 50 + assert bitrate == 192