diff --git a/README.md b/README.md index a4c1b05..7cbd1d1 100644 --- a/README.md +++ b/README.md @@ -61,8 +61,16 @@ Two consequences are worth stating up front. The incumbent it measures against i Autopilot's percentile recommender, not threshold HPA — beating threshold HPA is routine and proves little. And the predictability diagnostic is a cruder instrument than the field's: spectral entropy is the established forecastability measure, and a single lagged correlation -is known to be weaker, which is very likely why it failed to generalise (see Q13). Replacing -it with spectral entropy is the clearest improvement available to this project. +is known to be weaker. + +That reading suggested an obvious fix, and this project recommended it here. **The fix was +then measured and it does not work.** Spectral entropy lost to the incumbent measure in an +exploratory race and scored AUC 0.499 — chance — on a fresh cohort. A horizon-relative +spectral measure looked promising at AUC 0.700, was pre-registered, and collapsed to 0.526 +on data that had not generated it. Three training-free measures have now been tried on +individual serverless workloads and all three have failed, which is why the diagnostic is +scoped to fleet aggregates rather than patched. See Q13 and Q13b in +[`docs/EVAL.md`](docs/EVAL.md). ## Read the evaluation diff --git a/docs/EVAL.md b/docs/EVAL.md index 44a3383..3a1cec1 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -1300,3 +1300,73 @@ from config, but an apostrophe would have silently returned the wrong SKU's pric unexploitable in production only because of a CDN behaviour nobody had designed for. - **A Pareto summary statistic could rank two frontiers backwards**, and its test passed because the error cancelled between the two curves being compared. + +--- + +# Q13b — the lead did not replicate (2026-08-09) + +Pre-registered in `docs/PREREGISTRATION-Q13B.md` and committed, with the complete analysis +script, **before the confirmatory data existed**. The commit order is checkable in the git +history and is the only thing that makes the rest of this section worth reading. + +## Result + +One measure, one horizon, one test, on 80 workloads drawn fresh and explicitly disjoint from +the 43 the lead was found on. + +| | exploratory (2026-08-08) | confirmatory (2026-08-09) | +|---|---:|---:| +| low-frequency power, AUC @ 12 h | 0.700 | **0.526** | +| p | 0.027 (one of six tests) | **0.359** (one-sided, pre-registered) | +| n measurable | 34 | 69 | + +**Not confirmed.** The effect all but vanished. The design had roughly 0.9 power against a +true AUC of 0.700, so this is not a near miss on an underpowered test — 0.526 is what a +measure with essentially no signal looks like. + +The honest reading is the boring one: **0.700 was the winner's curse.** It was the largest of +six AUCs computed on 34 workloads, and the largest of six noisy estimates is biased upward by +construction. Regression to the mean did the rest. + +## Secondary, and deliberately not acted on + +On the same fresh cohort, daily autocorrelation scored **AUC 0.601** and spectral +predictability **0.499**. + +Autocorrelation scoring highest is exactly the sort of result that invites a second bite: +declare it vindicated, rebuild the diagnostic around it, quote 0.601. That is forbidden here +and the pre-registration says so in advance. These are descriptive numbers carrying no alpha, +computed on n = 69, where the permutation null from the exploratory run put the 95% ceiling +near 0.68 — so 0.601 is inside the noise band and establishes nothing. **No claim is made +from it, and the shipped diagnostic does not change.** + +## What this closes + +- **The low-frequency power measure is refuted**, not merely unconfirmed. It had one + well-powered pre-registered chance and did not take it. +- **Spectral entropy is refuted twice over.** It lost the exploratory race (AUC 0.396 and + 0.456) and scored 0.499 — chance, to three decimals — on fresh data. +- **No training-free measure tested predicts whether forecasting pays on individual + serverless workloads.** Daily autocorrelation, spectral entropy, and horizon-relative + spectral power have all now been tried and all have failed on that population. + +The diagnostic therefore stays exactly where Q13 left it: **valid on fleet-aggregate demand, +where it calls 27 of 28 cells correctly, and scoped out of everything else on the page a +visitor reads.** That scope is now supported by two independent failures to extend it rather +than one. + +## A prediction of ours that was wrong, recorded as such + +After the literature review on 2026-08-08 this project recommended replacing daily +autocorrelation with spectral entropy, calling it "the clearest improvement available". The +forecastability literature does support spectral measures in general, and the reasoning was +sound in the abstract. **On this problem it was wrong**, and measured twice to be wrong. The +recommendation is withdrawn from the README. + +## Added to the negatives ledger + +- **The Q13 replacement measure failed to replicate**, 0.700 → 0.526, against a + pre-registered hypothesis with ~0.9 power. +- **A promising exploratory AUC was the winner's curse**, and pre-registration is what caught + it rather than hindsight. +- **This project publicly recommended an improvement that its own next experiment refuted.** diff --git a/docs/PREREGISTRATION-Q13B.md b/docs/PREREGISTRATION-Q13B.md new file mode 100644 index 0000000..f387ab1 --- /dev/null +++ b/docs/PREREGISTRATION-Q13B.md @@ -0,0 +1,73 @@ +# Pre-registration — Q13b: does low-frequency power predict whether forecasting pays? + +**Written and committed before the confirmatory data was generated.** The commit that adds +this file also adds `scripts/evaluate_q13b.py`, which is the complete analysis. Nothing below +was chosen after seeing a result; the git history is the evidence for that claim, and it is +the only evidence worth anything. + +## Why a second test is needed + +The exploratory run on 2026-08-08 raced three training-free measures against whether +forecasting paid, over 43 serverless workloads at two commitment windows. The share of +spectral power at periods longer than the commitment window reached **AUC 0.700 at twelve +hours, permutation p = 0.027**. + +That is six tests (three measures x two windows). Bonferroni wants p < 0.0083 and 0.027 does +not clear it, so the result was recorded as a lead and deliberately not shipped. Reporting +the largest of six AUCs at its uncorrected p-value is how findings get manufactured. + +A lead becomes a finding by surviving **one** pre-specified test on data that did not +generate it. That is what this is. + +## The hypothesis, singular + +> **H1.** On individual Azure Functions workloads, at a **twelve-hour** commitment, +> `low_frequency_power_fraction` is positively associated with whether forecasting beats a +> trailing-percentile commitment on total economic cost. +> +> **H0.** No association: AUC = 0.5. + +One measure. One horizon. One test. Everything else was fixed before running. + +## Everything fixed in advance + +| Decision | Value | Why fixed now | +|---|---|---| +| Measure | `low_frequency_power_fraction(values, window_steps)` | The lead; no other measure is under test | +| Horizon | **12 hours** only | Where the lead appeared; testing both windows would restore the multiplicity problem | +| Outcome | Forecasting pays = ≥1 strict economic win across the 4 quantiles | Identical definition to the exploratory run | +| Scoring | Total economic cost at `C_u/C_o = q/(1-q)` | The newsvendor objective, as in Q4 | +| Forecaster | `SeasonalNaiveForecaster` | As in the exploratory run | +| Quantiles | 0.80, 0.90, 0.95, 0.99 | As in the exploratory run | +| Statistic | ROC AUC | Threshold-free; no cutoff to fit | +| Test | One-sided permutation, 20,000 draws, **α = 0.05** | One-sided because H1 predicts a direction | +| Sample size | **n = 80** | Power calculation below | +| Cohort | Fresh draw, seed `20260809`, **disjoint from the 43 exploratory workloads** | A confirmatory test on the generating data is not a test | +| Sampling | Uniform at random among usable workloads — **no stratification** | The exploratory run oversampled the borderline band; stratifying on the predictor under test would bias the AUC | +| Exclusions | Same `usable()` filters; workloads where all 4 quantiles tie are dropped as non-measurements | Fixed before running, as in the exploratory script | + +### Power + +Under H0 with a roughly balanced split at n = 80, the standard error of the AUC is about +`sqrt((n1+n2+1) / (12*n1*n2))` ≈ 0.065, so rejection needs an observed AUC above roughly +0.607. If the true AUC is the 0.700 the exploratory run suggested, power is approximately +**0.9**. If the true effect is appreciably smaller than that, this design will not detect it, +and a null result should be read as "not the effect size we thought", not "no effect". + +## What counts as what + +- **AUC > 0.5 with one-sided p < 0.05 → confirmed.** The measure predicts, and the diagnostic + can be rebuilt on it. +- **p ≥ 0.05 → not confirmed.** The lead does not replicate, it is recorded as refuted, and + the shipped diagnostic keeps daily autocorrelation with its existing scope caveat. +- **AUC < 0.5 → refuted**, whatever the p-value. + +There is no third option, no re-slicing by band, and no switching to the six-hour window if +twelve is disappointing. Any further analysis is exploratory and will be labelled as such. + +## Secondary, explicitly not the test + +Daily autocorrelation and spectral predictability are computed on the same fresh cohort and +reported alongside, so the three can be compared on data none of them has seen. **These are +descriptive.** They carry no α, they cannot confirm anything, and the outcome of H1 does not +depend on them. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index b82ab81..777cef9 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -54,6 +54,11 @@ it measured is narrower and more useful: ignoring the diagnostic entirely — with a non-monotone relationship that no version of the rule predicted. Only the bottom of the range transfers: below ~0.20, forecasting failed to pay on both populations. +5. **Q13b: the replacement does not work either.** Spectral entropy, the field's standard + forecastability measure, scored at chance on a fresh cohort. A horizon-relative spectral + measure reached AUC 0.700 exploratorily, was pre-registered, and fell to 0.526 — the + winner's curse, caught by pre-registration rather than by hindsight. Three measures + tried, three failed, so the diagnostic is scoped rather than patched. The shipped artifact is therefore a **diagnostic that tells you whether to build a forecaster at all**, scoped to aggregated demand and stating that scope on its own results page. That is diff --git a/scripts/build_snapshot.py b/scripts/build_snapshot.py index 72b5df1..b8b0d29 100644 --- a/scripts/build_snapshot.py +++ b/scripts/build_snapshot.py @@ -252,8 +252,13 @@ def findings() -> list[Finding]: "(AUC 0.396 and 0.456). The one candidate showing signal — the share of " "spectral power slower than the commitment window — reaches AUC 0.700 at 12 " "hours but does not survive Bonferroni across the six tests run, so it is " - "recorded as a lead rather than promoted. What survives: below r=0.20, " - "forecasting failed to pay on every population tested." + "recorded as a lead rather than promoted. That lead was then pre-registered " + "and tested on 80 fresh workloads disjoint from the ones that produced it: " + "it fell from AUC 0.700 to 0.526 (one-sided p=0.36) against a design with " + "roughly 0.9 power, so it was the winner's curse and is refuted. Spectral " + "entropy scored 0.499 on the same fresh cohort. Three training-free measures " + "tried, three failed. What survives: below r=0.20, forecasting failed to pay " + "on every population tested." ), ), ] diff --git a/scripts/evaluate_q13b.py b/scripts/evaluate_q13b.py new file mode 100644 index 0000000..b82f120 --- /dev/null +++ b/scripts/evaluate_q13b.py @@ -0,0 +1,164 @@ +"""Q13b — the confirmatory test pre-registered in `docs/PREREGISTRATION-Q13B.md`. + +One measure, one horizon, one test, on a cohort disjoint from the one that produced the +lead. This script is committed together with the pre-registration and before the +confirmatory data existed; the git history is what makes that claim checkable. + +Read the pre-registration first. Nothing here may be changed after a result is seen without +saying so loudly in `docs/EVAL.md`. + +Run: `python scripts/evaluate_q13b.py` +""" + +from __future__ import annotations + +import argparse + +import numpy as np +from compare_predictability import auc, measures_for # type: ignore[import-not-found] +from evaluate_threshold import ( # type: ignore[import-not-found] + BIN_SECONDS, + build_candidates, + stratify, + strict_dominance, +) + +# --- everything below is pre-registered; see docs/PREREGISTRATION-Q13B.md ------------- +MEASURE = "low-freq power" +WINDOW_HOURS = 12 +SAMPLE_SIZE = 80 +ALPHA = 0.05 +PERMUTATIONS = 20_000 +CONFIRMATORY_SEED = 20260809 + +#: The exploratory run these must not overlap with. +EXPLORATORY_COHORT = 400 +EXPLORATORY_SAMPLE = 60 +EXPLORATORY_SEED = 20260808 + +SECONDARY = ("daily autocorr", "spectral predictability") + + +def exploratory_names() -> set[str]: + """Exactly the workloads the lead was found on, so they can be excluded.""" + seen = stratify( + build_candidates(EXPLORATORY_COHORT, EXPLORATORY_SEED), + EXPLORATORY_SAMPLE, + EXPLORATORY_SEED, + ) + return {candidate.workload.name for candidate in seen} + + +def one_sided_permutation_p( + scores: list[float], labels: list[bool], observed: float, seed: int +) -> float: + """P(AUC >= observed | labels shuffled). One-sided: H1 predicts a direction.""" + rng = np.random.default_rng(seed) + shuffled = np.asarray(labels, dtype=bool) + hits = 0 + for _ in range(PERMUTATIONS): + if auc(scores, list(rng.permutation(shuffled))) >= observed: + hits += 1 + return (hits + 1) / (PERMUTATIONS + 1) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--cohort", type=int, default=900, help="workloads to load and filter") + args = parser.parse_args() + + window_steps = WINDOW_HOURS * 3600 // BIN_SECONDS + + print("# Q13b — confirmatory test\n") + print( + f"Pre-registered in `docs/PREREGISTRATION-Q13B.md`. Measure: **{MEASURE}**. " + f"Horizon: **{WINDOW_HOURS} h**. One-sided permutation test, " + f"alpha = {ALPHA}, {PERMUTATIONS:,} draws. Target n = {SAMPLE_SIZE}.\n" + ) + + excluded = exploratory_names() + fresh = [ + candidate + for candidate in build_candidates(args.cohort, CONFIRMATORY_SEED) + if candidate.workload.name not in excluded + ] + rng = np.random.default_rng(CONFIRMATORY_SEED) + if len(fresh) > SAMPLE_SIZE: + index = rng.choice(len(fresh), size=SAMPLE_SIZE, replace=False) + fresh = [fresh[int(i)] for i in index] + print( + f"Excluded {len(excluded)} exploratory workloads. " + f"Confirmatory cohort: {len(fresh)} workloads, sampled uniformly at random " + "with no stratification.\n" + ) + + rows: list[tuple[str, dict[str, float], bool]] = [] + all_tie = 0 + for candidate in fresh: + workload = candidate.workload + wins, ties = strict_dominance(workload, window_steps) + if ties == 4: + all_tie += 1 + continue + rows.append( + ( + workload.name, + measures_for(workload.series.values, workload.day_steps, window_steps), + wins > 0, + ) + ) + + labels = [paid for _, _, paid in rows] + print( + f"Measurable: {len(rows)} ({sum(labels)} where forecasting paid, " + f"{len(labels) - sum(labels)} where it did not). " + f"Dropped as all-tie non-measurements: {all_tie}.\n" + ) + if len(set(labels)) < 2 or len(rows) < 20: + print("**Test not evaluable** — the cohort is degenerate. Reported, not worked around.") + return + + scores = [measures[MEASURE] for _, measures, _ in rows] + observed = auc(scores, labels) + p_value = one_sided_permutation_p(scores, labels, observed, CONFIRMATORY_SEED) + confirmed = observed > 0.5 and p_value < ALPHA + + print("## The pre-registered test\n") + print("| measure | AUC | one-sided p | alpha | verdict |") + print("|---|---:|---:|---:|:--:|") + print( + f"| {MEASURE} | {observed:.3f} | {p_value:.4f} | {ALPHA} | " + f"**{'CONFIRMED' if confirmed else 'NOT CONFIRMED'}** |" + ) + print() + if confirmed: + print( + f"The lead replicates on data that did not generate it. AUC {observed:.3f} at a " + f"{WINDOW_HOURS}-hour commitment, p = {p_value:.4f} against a single " + "pre-registered hypothesis." + ) + elif observed <= 0.5: + print( + f"**Refuted.** AUC {observed:.3f} is at or below chance, so the measure carries " + "no usable signal on a fresh cohort regardless of the p-value." + ) + else: + print( + f"**Not confirmed.** AUC {observed:.3f} points the right way but p = {p_value:.4f} " + f"does not clear {ALPHA}. On the pre-registered rule this is a null result and the " + "shipped diagnostic does not change." + ) + + print("\n## Secondary — descriptive, not the test\n") + print("| measure | AUC on the same fresh cohort |") + print("|---|---:|") + for key in SECONDARY: + print(f"| {key} | {auc([m[key] for _, m, _ in rows], labels):.3f} |") + print( + "\nThese carry no alpha and cannot confirm anything. They are here so the three " + "measures can be compared on data none of them has seen." + ) + + +if __name__ == "__main__": + main()