Q13b: the pre-registered lead did not replicate

README.md

@@ -61,8 +61,16 @@ Two consequences are worth stating up front. The incumbent it measures against i
 Autopilot's percentile recommender, not threshold HPA — beating threshold HPA is routine and
 proves little. And the predictability diagnostic is a cruder instrument than the field's:
 spectral entropy is the established forecastability measure, and a single lagged correlation
-is known to be weaker, which is very likely why it failed to generalise (see Q13). Replacing
-it with spectral entropy is the clearest improvement available to this project.
+is known to be weaker.
+
+That reading suggested an obvious fix, and this project recommended it here. **The fix was
+then measured and it does not work.** Spectral entropy lost to the incumbent measure in an
+exploratory race and scored AUC 0.499 — chance — on a fresh cohort. A horizon-relative
+spectral measure looked promising at AUC 0.700, was pre-registered, and collapsed to 0.526
+on data that had not generated it. Three training-free measures have now been tried on
+individual serverless workloads and all three have failed, which is why the diagnostic is
+scoped to fleet aggregates rather than patched. See Q13 and Q13b in
+[`docs/EVAL.md`](docs/EVAL.md).
 
 ## Read the evaluation
 

docs/EVAL.md

@@ -1300,3 +1300,73 @@ from config, but an apostrophe would have silently returned the wrong SKU's pric
   unexploitable in production only because of a CDN behaviour nobody had designed for.
 - **A Pareto summary statistic could rank two frontiers backwards**, and its test passed
   because the error cancelled between the two curves being compared.
+
+---
+
+# Q13b — the lead did not replicate (2026-08-09)
+
+Pre-registered in `docs/PREREGISTRATION-Q13B.md` and committed, with the complete analysis
+script, **before the confirmatory data existed**. The commit order is checkable in the git
+history and is the only thing that makes the rest of this section worth reading.
+
+## Result
+
+One measure, one horizon, one test, on 80 workloads drawn fresh and explicitly disjoint from
+the 43 the lead was found on.
+
+| | exploratory (2026-08-08) | confirmatory (2026-08-09) |
+|---|---:|---:|
+| low-frequency power, AUC @ 12 h | 0.700 | **0.526** |
+| p | 0.027 (one of six tests) | **0.359** (one-sided, pre-registered) |
+| n measurable | 34 | 69 |
+
+**Not confirmed.** The effect all but vanished. The design had roughly 0.9 power against a
+true AUC of 0.700, so this is not a near miss on an underpowered test — 0.526 is what a
+measure with essentially no signal looks like.
+
+The honest reading is the boring one: **0.700 was the winner's curse.** It was the largest of
+six AUCs computed on 34 workloads, and the largest of six noisy estimates is biased upward by
+construction. Regression to the mean did the rest.
+
+## Secondary, and deliberately not acted on
+
+On the same fresh cohort, daily autocorrelation scored **AUC 0.601** and spectral
+predictability **0.499**.
+
+Autocorrelation scoring highest is exactly the sort of result that invites a second bite:
+declare it vindicated, rebuild the diagnostic around it, quote 0.601. That is forbidden here
+and the pre-registration says so in advance. These are descriptive numbers carrying no alpha,
+computed on n = 69, where the permutation null from the exploratory run put the 95% ceiling
+near 0.68 — so 0.601 is inside the noise band and establishes nothing. **No claim is made
+from it, and the shipped diagnostic does not change.**
+
+## What this closes
+
+- **The low-frequency power measure is refuted**, not merely unconfirmed. It had one
+  well-powered pre-registered chance and did not take it.
+- **Spectral entropy is refuted twice over.** It lost the exploratory race (AUC 0.396 and
+  0.456) and scored 0.499 — chance, to three decimals — on fresh data.
+- **No training-free measure tested predicts whether forecasting pays on individual
+  serverless workloads.** Daily autocorrelation, spectral entropy, and horizon-relative
+  spectral power have all now been tried and all have failed on that population.
+
+The diagnostic therefore stays exactly where Q13 left it: **valid on fleet-aggregate demand,
+where it calls 27 of 28 cells correctly, and scoped out of everything else on the page a
+visitor reads.** That scope is now supported by two independent failures to extend it rather
+than one.
+
+## A prediction of ours that was wrong, recorded as such
+
+After the literature review on 2026-08-08 this project recommended replacing daily
+autocorrelation with spectral entropy, calling it "the clearest improvement available". The
+forecastability literature does support spectral measures in general, and the reasoning was
+sound in the abstract. **On this problem it was wrong**, and measured twice to be wrong. The
+recommendation is withdrawn from the README.
+
+## Added to the negatives ledger
+
+- **The Q13 replacement measure failed to replicate**, 0.700 → 0.526, against a
+  pre-registered hypothesis with ~0.9 power.
+- **A promising exploratory AUC was the winner's curse**, and pre-registration is what caught
+  it rather than hindsight.
+- **This project publicly recommended an improvement that its own next experiment refuted.**

docs/PREREGISTRATION-Q13B.md

@@ -0,0 +1,73 @@
+# Pre-registration — Q13b: does low-frequency power predict whether forecasting pays?
+
+**Written and committed before the confirmatory data was generated.** The commit that adds
+this file also adds `scripts/evaluate_q13b.py`, which is the complete analysis. Nothing below
+was chosen after seeing a result; the git history is the evidence for that claim, and it is
+the only evidence worth anything.
+
+## Why a second test is needed
+
+The exploratory run on 2026-08-08 raced three training-free measures against whether
+forecasting paid, over 43 serverless workloads at two commitment windows. The share of
+spectral power at periods longer than the commitment window reached **AUC 0.700 at twelve
+hours, permutation p = 0.027**.
+
+That is six tests (three measures x two windows). Bonferroni wants p < 0.0083 and 0.027 does
+not clear it, so the result was recorded as a lead and deliberately not shipped. Reporting
+the largest of six AUCs at its uncorrected p-value is how findings get manufactured.
+
+A lead becomes a finding by surviving **one** pre-specified test on data that did not
+generate it. That is what this is.
+
+## The hypothesis, singular
+
+> **H1.** On individual Azure Functions workloads, at a **twelve-hour** commitment,
+> `low_frequency_power_fraction` is positively associated with whether forecasting beats a
+> trailing-percentile commitment on total economic cost.
+>
+> **H0.** No association: AUC = 0.5.
+
+One measure. One horizon. One test. Everything else was fixed before running.
+
+## Everything fixed in advance
+
+| Decision | Value | Why fixed now |
+|---|---|---|
+| Measure | `low_frequency_power_fraction(values, window_steps)` | The lead; no other measure is under test |
+| Horizon | **12 hours** only | Where the lead appeared; testing both windows would restore the multiplicity problem |
+| Outcome | Forecasting pays = ≥1 strict economic win across the 4 quantiles | Identical definition to the exploratory run |
+| Scoring | Total economic cost at `C_u/C_o = q/(1-q)` | The newsvendor objective, as in Q4 |
+| Forecaster | `SeasonalNaiveForecaster` | As in the exploratory run |
+| Quantiles | 0.80, 0.90, 0.95, 0.99 | As in the exploratory run |
+| Statistic | ROC AUC | Threshold-free; no cutoff to fit |
+| Test | One-sided permutation, 20,000 draws, **α = 0.05** | One-sided because H1 predicts a direction |
+| Sample size | **n = 80** | Power calculation below |
+| Cohort | Fresh draw, seed `20260809`, **disjoint from the 43 exploratory workloads** | A confirmatory test on the generating data is not a test |
+| Sampling | Uniform at random among usable workloads — **no stratification** | The exploratory run oversampled the borderline band; stratifying on the predictor under test would bias the AUC |
+| Exclusions | Same `usable()` filters; workloads where all 4 quantiles tie are dropped as non-measurements | Fixed before running, as in the exploratory script |
+
+### Power
+
+Under H0 with a roughly balanced split at n = 80, the standard error of the AUC is about
+`sqrt((n1+n2+1) / (12*n1*n2))` ≈ 0.065, so rejection needs an observed AUC above roughly
+0.607. If the true AUC is the 0.700 the exploratory run suggested, power is approximately
+**0.9**. If the true effect is appreciably smaller than that, this design will not detect it,
+and a null result should be read as "not the effect size we thought", not "no effect".
+
+## What counts as what
+
+- **AUC > 0.5 with one-sided p < 0.05 → confirmed.** The measure predicts, and the diagnostic
+  can be rebuilt on it.
+- **p ≥ 0.05 → not confirmed.** The lead does not replicate, it is recorded as refuted, and
+  the shipped diagnostic keeps daily autocorrelation with its existing scope caveat.
+- **AUC < 0.5 → refuted**, whatever the p-value.
+
+There is no third option, no re-slicing by band, and no switching to the six-hour window if
+twelve is disappointing. Any further analysis is exploratory and will be labelled as such.
+
+## Secondary, explicitly not the test
+
+Daily autocorrelation and spectral predictability are computed on the same fresh cohort and
+reported alongside, so the three can be compared on data none of them has seen. **These are
+descriptive.** They carry no α, they cannot confirm anything, and the outcome of H1 does not
+depend on them.

docs/ROADMAP.md

@@ -54,6 +54,11 @@ it measured is narrower and more useful:
    ignoring the diagnostic entirely — with a non-monotone relationship that no version of
    the rule predicted. Only the bottom of the range transfers: below ~0.20, forecasting
    failed to pay on both populations.
+5. **Q13b: the replacement does not work either.** Spectral entropy, the field's standard
+   forecastability measure, scored at chance on a fresh cohort. A horizon-relative spectral
+   measure reached AUC 0.700 exploratorily, was pre-registered, and fell to 0.526 — the
+   winner's curse, caught by pre-registration rather than by hindsight. Three measures
+   tried, three failed, so the diagnostic is scoped rather than patched.
 
 The shipped artifact is therefore a **diagnostic that tells you whether to build a forecaster
 at all**, scoped to aggregated demand and stating that scope on its own results page. That is

scripts/build_snapshot.py

@@ -252,8 +252,13 @@ def findings() -> list[Finding]:
                 "(AUC 0.396 and 0.456). The one candidate showing signal — the share of "
                 "spectral power slower than the commitment window — reaches AUC 0.700 at 12 "
                 "hours but does not survive Bonferroni across the six tests run, so it is "
-                "recorded as a lead rather than promoted. What survives: below r=0.20, "
-                "forecasting failed to pay on every population tested."
+                "recorded as a lead rather than promoted. That lead was then pre-registered "
+                "and tested on 80 fresh workloads disjoint from the ones that produced it: "
+                "it fell from AUC 0.700 to 0.526 (one-sided p=0.36) against a design with "
+                "roughly 0.9 power, so it was the winner's curse and is refuted. Spectral "
+                "entropy scored 0.499 on the same fresh cohort. Three training-free measures "
+                "tried, three failed. What survives: below r=0.20, forecasting failed to pay "
+                "on every population tested."
             ),
         ),
     ]

scripts/evaluate_q13b.py

@@ -0,0 +1,164 @@
+"""Q13b — the confirmatory test pre-registered in `docs/PREREGISTRATION-Q13B.md`.
+
+One measure, one horizon, one test, on a cohort disjoint from the one that produced the
+lead. This script is committed together with the pre-registration and before the
+confirmatory data existed; the git history is what makes that claim checkable.
+
+Read the pre-registration first. Nothing here may be changed after a result is seen without
+saying so loudly in `docs/EVAL.md`.
+
+Run: `python scripts/evaluate_q13b.py`
+"""
+
+from __future__ import annotations
+
+import argparse
+
+import numpy as np
+from compare_predictability import auc, measures_for  # type: ignore[import-not-found]
+from evaluate_threshold import (  # type: ignore[import-not-found]
+    BIN_SECONDS,
+    build_candidates,
+    stratify,
+    strict_dominance,
+)
+
+# --- everything below is pre-registered; see docs/PREREGISTRATION-Q13B.md -------------
+MEASURE = "low-freq power"
+WINDOW_HOURS = 12
+SAMPLE_SIZE = 80
+ALPHA = 0.05
+PERMUTATIONS = 20_000
+CONFIRMATORY_SEED = 20260809
+
+#: The exploratory run these must not overlap with.
+EXPLORATORY_COHORT = 400
+EXPLORATORY_SAMPLE = 60
+EXPLORATORY_SEED = 20260808
+
+SECONDARY = ("daily autocorr", "spectral predictability")
+
+
+def exploratory_names() -> set[str]:
+    """Exactly the workloads the lead was found on, so they can be excluded."""
+    seen = stratify(
+        build_candidates(EXPLORATORY_COHORT, EXPLORATORY_SEED),
+        EXPLORATORY_SAMPLE,
+        EXPLORATORY_SEED,
+    )
+    return {candidate.workload.name for candidate in seen}
+
+
+def one_sided_permutation_p(
+    scores: list[float], labels: list[bool], observed: float, seed: int
+) -> float:
+    """P(AUC >= observed | labels shuffled). One-sided: H1 predicts a direction."""
+    rng = np.random.default_rng(seed)
+    shuffled = np.asarray(labels, dtype=bool)
+    hits = 0
+    for _ in range(PERMUTATIONS):
+        if auc(scores, list(rng.permutation(shuffled))) >= observed:
+            hits += 1
+    return (hits + 1) / (PERMUTATIONS + 1)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--cohort", type=int, default=900, help="workloads to load and filter")
+    args = parser.parse_args()
+
+    window_steps = WINDOW_HOURS * 3600 // BIN_SECONDS
+
+    print("# Q13b — confirmatory test\n")
+    print(
+        f"Pre-registered in `docs/PREREGISTRATION-Q13B.md`. Measure: **{MEASURE}**. "
+        f"Horizon: **{WINDOW_HOURS} h**. One-sided permutation test, "
+        f"alpha = {ALPHA}, {PERMUTATIONS:,} draws. Target n = {SAMPLE_SIZE}.\n"
+    )
+
+    excluded = exploratory_names()
+    fresh = [
+        candidate
+        for candidate in build_candidates(args.cohort, CONFIRMATORY_SEED)
+        if candidate.workload.name not in excluded
+    ]
+    rng = np.random.default_rng(CONFIRMATORY_SEED)
+    if len(fresh) > SAMPLE_SIZE:
+        index = rng.choice(len(fresh), size=SAMPLE_SIZE, replace=False)
+        fresh = [fresh[int(i)] for i in index]
+    print(
+        f"Excluded {len(excluded)} exploratory workloads. "
+        f"Confirmatory cohort: {len(fresh)} workloads, sampled uniformly at random "
+        "with no stratification.\n"
+    )
+
+    rows: list[tuple[str, dict[str, float], bool]] = []
+    all_tie = 0
+    for candidate in fresh:
+        workload = candidate.workload
+        wins, ties = strict_dominance(workload, window_steps)
+        if ties == 4:
+            all_tie += 1
+            continue
+        rows.append(
+            (
+                workload.name,
+                measures_for(workload.series.values, workload.day_steps, window_steps),
+                wins > 0,
+            )
+        )
+
+    labels = [paid for _, _, paid in rows]
+    print(
+        f"Measurable: {len(rows)} ({sum(labels)} where forecasting paid, "
+        f"{len(labels) - sum(labels)} where it did not). "
+        f"Dropped as all-tie non-measurements: {all_tie}.\n"
+    )
+    if len(set(labels)) < 2 or len(rows) < 20:
+        print("**Test not evaluable** — the cohort is degenerate. Reported, not worked around.")
+        return
+
+    scores = [measures[MEASURE] for _, measures, _ in rows]
+    observed = auc(scores, labels)
+    p_value = one_sided_permutation_p(scores, labels, observed, CONFIRMATORY_SEED)
+    confirmed = observed > 0.5 and p_value < ALPHA
+
+    print("## The pre-registered test\n")
+    print("| measure | AUC | one-sided p | alpha | verdict |")
+    print("|---|---:|---:|---:|:--:|")
+    print(
+        f"| {MEASURE} | {observed:.3f} | {p_value:.4f} | {ALPHA} | "
+        f"**{'CONFIRMED' if confirmed else 'NOT CONFIRMED'}** |"
+    )
+    print()
+    if confirmed:
+        print(
+            f"The lead replicates on data that did not generate it. AUC {observed:.3f} at a "
+            f"{WINDOW_HOURS}-hour commitment, p = {p_value:.4f} against a single "
+            "pre-registered hypothesis."
+        )
+    elif observed <= 0.5:
+        print(
+            f"**Refuted.** AUC {observed:.3f} is at or below chance, so the measure carries "
+            "no usable signal on a fresh cohort regardless of the p-value."
+        )
+    else:
+        print(
+            f"**Not confirmed.** AUC {observed:.3f} points the right way but p = {p_value:.4f} "
+            f"does not clear {ALPHA}. On the pre-registered rule this is a null result and the "
+            "shipped diagnostic does not change."
+        )
+
+    print("\n## Secondary — descriptive, not the test\n")
+    print("| measure | AUC on the same fresh cohort |")
+    print("|---|---:|")
+    for key in SECONDARY:
+        print(f"| {key} | {auc([m[key] for _, m, _ in rows], labels):.3f} |")
+    print(
+        "\nThese carry no alpha and cannot confirm anything. They are here so the three "
+        "measures can be compared on data none of them has seen."
+    )
+
+
+if __name__ == "__main__":
+    main()