From ed3aa414910dc1077ebf4b026550b3f6a320bd6e Mon Sep 17 00:00:00 2001 From: "@jaydenOoOo" Date: Sun, 9 Aug 2026 15:43:26 +0800 Subject: [PATCH 1/2] Pre-register the Q13b confirmatory test, before running it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The low-frequency power lead reached AUC 0.700 at a twelve-hour commitment with permutation p = 0.027, which does not survive Bonferroni across the six tests that produced it. A lead becomes a finding by surviving one pre-specified test on data that did not generate it, so this commits the hypothesis and the complete analysis before the confirmatory data exists. The commit order is the only evidence that claim is worth anything. One measure, one horizon, one test. Twelve hours only — testing both windows would restore the multiplicity problem the lead already failed. Alpha 0.05, one-sided because the hypothesis predicts a direction, 20,000 permutations. Target n = 80, chosen for roughly 0.9 power against the AUC the exploratory run suggested; a null at this size means the effect is smaller than we thought, not that it is absent. The cohort is drawn fresh under a new seed and explicitly excludes the 43 workloads the lead was found on, regenerated deterministically rather than hard-coded. Sampling is uniform with no stratification: the exploratory run oversampled the borderline band, and stratifying on the predictor under test would bias the statistic being tested. The decision rule is written down with no third option. No re-slicing by band, and no falling back to the six-hour window if twelve disappoints. Daily autocorrelation and spectral predictability are computed on the same cohort as descriptive comparisons that carry no alpha and cannot confirm anything. --- docs/PREREGISTRATION-Q13B.md | 73 ++++++++++++++++ scripts/evaluate_q13b.py | 164 +++++++++++++++++++++++++++++++++++ 2 files changed, 237 insertions(+) create mode 100644 docs/PREREGISTRATION-Q13B.md create mode 100644 scripts/evaluate_q13b.py diff --git a/docs/PREREGISTRATION-Q13B.md b/docs/PREREGISTRATION-Q13B.md new file mode 100644 index 0000000..f387ab1 --- /dev/null +++ b/docs/PREREGISTRATION-Q13B.md @@ -0,0 +1,73 @@ +# Pre-registration — Q13b: does low-frequency power predict whether forecasting pays? + +**Written and committed before the confirmatory data was generated.** The commit that adds +this file also adds `scripts/evaluate_q13b.py`, which is the complete analysis. Nothing below +was chosen after seeing a result; the git history is the evidence for that claim, and it is +the only evidence worth anything. + +## Why a second test is needed + +The exploratory run on 2026-08-08 raced three training-free measures against whether +forecasting paid, over 43 serverless workloads at two commitment windows. The share of +spectral power at periods longer than the commitment window reached **AUC 0.700 at twelve +hours, permutation p = 0.027**. + +That is six tests (three measures x two windows). Bonferroni wants p < 0.0083 and 0.027 does +not clear it, so the result was recorded as a lead and deliberately not shipped. Reporting +the largest of six AUCs at its uncorrected p-value is how findings get manufactured. + +A lead becomes a finding by surviving **one** pre-specified test on data that did not +generate it. That is what this is. + +## The hypothesis, singular + +> **H1.** On individual Azure Functions workloads, at a **twelve-hour** commitment, +> `low_frequency_power_fraction` is positively associated with whether forecasting beats a +> trailing-percentile commitment on total economic cost. +> +> **H0.** No association: AUC = 0.5. + +One measure. One horizon. One test. Everything else was fixed before running. + +## Everything fixed in advance + +| Decision | Value | Why fixed now | +|---|---|---| +| Measure | `low_frequency_power_fraction(values, window_steps)` | The lead; no other measure is under test | +| Horizon | **12 hours** only | Where the lead appeared; testing both windows would restore the multiplicity problem | +| Outcome | Forecasting pays = ≥1 strict economic win across the 4 quantiles | Identical definition to the exploratory run | +| Scoring | Total economic cost at `C_u/C_o = q/(1-q)` | The newsvendor objective, as in Q4 | +| Forecaster | `SeasonalNaiveForecaster` | As in the exploratory run | +| Quantiles | 0.80, 0.90, 0.95, 0.99 | As in the exploratory run | +| Statistic | ROC AUC | Threshold-free; no cutoff to fit | +| Test | One-sided permutation, 20,000 draws, **α = 0.05** | One-sided because H1 predicts a direction | +| Sample size | **n = 80** | Power calculation below | +| Cohort | Fresh draw, seed `20260809`, **disjoint from the 43 exploratory workloads** | A confirmatory test on the generating data is not a test | +| Sampling | Uniform at random among usable workloads — **no stratification** | The exploratory run oversampled the borderline band; stratifying on the predictor under test would bias the AUC | +| Exclusions | Same `usable()` filters; workloads where all 4 quantiles tie are dropped as non-measurements | Fixed before running, as in the exploratory script | + +### Power + +Under H0 with a roughly balanced split at n = 80, the standard error of the AUC is about +`sqrt((n1+n2+1) / (12*n1*n2))` ≈ 0.065, so rejection needs an observed AUC above roughly +0.607. If the true AUC is the 0.700 the exploratory run suggested, power is approximately +**0.9**. If the true effect is appreciably smaller than that, this design will not detect it, +and a null result should be read as "not the effect size we thought", not "no effect". + +## What counts as what + +- **AUC > 0.5 with one-sided p < 0.05 → confirmed.** The measure predicts, and the diagnostic + can be rebuilt on it. +- **p ≥ 0.05 → not confirmed.** The lead does not replicate, it is recorded as refuted, and + the shipped diagnostic keeps daily autocorrelation with its existing scope caveat. +- **AUC < 0.5 → refuted**, whatever the p-value. + +There is no third option, no re-slicing by band, and no switching to the six-hour window if +twelve is disappointing. Any further analysis is exploratory and will be labelled as such. + +## Secondary, explicitly not the test + +Daily autocorrelation and spectral predictability are computed on the same fresh cohort and +reported alongside, so the three can be compared on data none of them has seen. **These are +descriptive.** They carry no α, they cannot confirm anything, and the outcome of H1 does not +depend on them. diff --git a/scripts/evaluate_q13b.py b/scripts/evaluate_q13b.py new file mode 100644 index 0000000..b82f120 --- /dev/null +++ b/scripts/evaluate_q13b.py @@ -0,0 +1,164 @@ +"""Q13b — the confirmatory test pre-registered in `docs/PREREGISTRATION-Q13B.md`. + +One measure, one horizon, one test, on a cohort disjoint from the one that produced the +lead. This script is committed together with the pre-registration and before the +confirmatory data existed; the git history is what makes that claim checkable. + +Read the pre-registration first. Nothing here may be changed after a result is seen without +saying so loudly in `docs/EVAL.md`. + +Run: `python scripts/evaluate_q13b.py` +""" + +from __future__ import annotations + +import argparse + +import numpy as np +from compare_predictability import auc, measures_for # type: ignore[import-not-found] +from evaluate_threshold import ( # type: ignore[import-not-found] + BIN_SECONDS, + build_candidates, + stratify, + strict_dominance, +) + +# --- everything below is pre-registered; see docs/PREREGISTRATION-Q13B.md ------------- +MEASURE = "low-freq power" +WINDOW_HOURS = 12 +SAMPLE_SIZE = 80 +ALPHA = 0.05 +PERMUTATIONS = 20_000 +CONFIRMATORY_SEED = 20260809 + +#: The exploratory run these must not overlap with. +EXPLORATORY_COHORT = 400 +EXPLORATORY_SAMPLE = 60 +EXPLORATORY_SEED = 20260808 + +SECONDARY = ("daily autocorr", "spectral predictability") + + +def exploratory_names() -> set[str]: + """Exactly the workloads the lead was found on, so they can be excluded.""" + seen = stratify( + build_candidates(EXPLORATORY_COHORT, EXPLORATORY_SEED), + EXPLORATORY_SAMPLE, + EXPLORATORY_SEED, + ) + return {candidate.workload.name for candidate in seen} + + +def one_sided_permutation_p( + scores: list[float], labels: list[bool], observed: float, seed: int +) -> float: + """P(AUC >= observed | labels shuffled). One-sided: H1 predicts a direction.""" + rng = np.random.default_rng(seed) + shuffled = np.asarray(labels, dtype=bool) + hits = 0 + for _ in range(PERMUTATIONS): + if auc(scores, list(rng.permutation(shuffled))) >= observed: + hits += 1 + return (hits + 1) / (PERMUTATIONS + 1) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--cohort", type=int, default=900, help="workloads to load and filter") + args = parser.parse_args() + + window_steps = WINDOW_HOURS * 3600 // BIN_SECONDS + + print("# Q13b — confirmatory test\n") + print( + f"Pre-registered in `docs/PREREGISTRATION-Q13B.md`. Measure: **{MEASURE}**. " + f"Horizon: **{WINDOW_HOURS} h**. One-sided permutation test, " + f"alpha = {ALPHA}, {PERMUTATIONS:,} draws. Target n = {SAMPLE_SIZE}.\n" + ) + + excluded = exploratory_names() + fresh = [ + candidate + for candidate in build_candidates(args.cohort, CONFIRMATORY_SEED) + if candidate.workload.name not in excluded + ] + rng = np.random.default_rng(CONFIRMATORY_SEED) + if len(fresh) > SAMPLE_SIZE: + index = rng.choice(len(fresh), size=SAMPLE_SIZE, replace=False) + fresh = [fresh[int(i)] for i in index] + print( + f"Excluded {len(excluded)} exploratory workloads. " + f"Confirmatory cohort: {len(fresh)} workloads, sampled uniformly at random " + "with no stratification.\n" + ) + + rows: list[tuple[str, dict[str, float], bool]] = [] + all_tie = 0 + for candidate in fresh: + workload = candidate.workload + wins, ties = strict_dominance(workload, window_steps) + if ties == 4: + all_tie += 1 + continue + rows.append( + ( + workload.name, + measures_for(workload.series.values, workload.day_steps, window_steps), + wins > 0, + ) + ) + + labels = [paid for _, _, paid in rows] + print( + f"Measurable: {len(rows)} ({sum(labels)} where forecasting paid, " + f"{len(labels) - sum(labels)} where it did not). " + f"Dropped as all-tie non-measurements: {all_tie}.\n" + ) + if len(set(labels)) < 2 or len(rows) < 20: + print("**Test not evaluable** — the cohort is degenerate. Reported, not worked around.") + return + + scores = [measures[MEASURE] for _, measures, _ in rows] + observed = auc(scores, labels) + p_value = one_sided_permutation_p(scores, labels, observed, CONFIRMATORY_SEED) + confirmed = observed > 0.5 and p_value < ALPHA + + print("## The pre-registered test\n") + print("| measure | AUC | one-sided p | alpha | verdict |") + print("|---|---:|---:|---:|:--:|") + print( + f"| {MEASURE} | {observed:.3f} | {p_value:.4f} | {ALPHA} | " + f"**{'CONFIRMED' if confirmed else 'NOT CONFIRMED'}** |" + ) + print() + if confirmed: + print( + f"The lead replicates on data that did not generate it. AUC {observed:.3f} at a " + f"{WINDOW_HOURS}-hour commitment, p = {p_value:.4f} against a single " + "pre-registered hypothesis." + ) + elif observed <= 0.5: + print( + f"**Refuted.** AUC {observed:.3f} is at or below chance, so the measure carries " + "no usable signal on a fresh cohort regardless of the p-value." + ) + else: + print( + f"**Not confirmed.** AUC {observed:.3f} points the right way but p = {p_value:.4f} " + f"does not clear {ALPHA}. On the pre-registered rule this is a null result and the " + "shipped diagnostic does not change." + ) + + print("\n## Secondary — descriptive, not the test\n") + print("| measure | AUC on the same fresh cohort |") + print("|---|---:|") + for key in SECONDARY: + print(f"| {key} | {auc([m[key] for _, m, _ in rows], labels):.3f} |") + print( + "\nThese carry no alpha and cannot confirm anything. They are here so the three " + "measures can be compared on data none of them has seen." + ) + + +if __name__ == "__main__": + main() From e8dccf9c8b902d0eee2dbb566297247b70a02041 Mon Sep 17 00:00:00 2001 From: "@jaydenOoOo" Date: Sun, 9 Aug 2026 15:49:13 +0800 Subject: [PATCH 2/2] Q13b: the lead did not replicate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the test pre-registered in the previous commit. The horizon-relative spectral measure fell from AUC 0.700 to 0.526 on 80 fresh workloads disjoint from the ones that produced it, one-sided p = 0.36 against a design with roughly 0.9 power. Not a near miss on an underpowered test — 0.526 is what no signal looks like. The honest reading is the boring one. 0.700 was the largest of six AUCs computed on 34 workloads, and the largest of six noisy estimates is biased upward by construction. Pre-registration is what caught that rather than hindsight, which is the entire reason it was written first. Daily autocorrelation scored 0.601 on the same fresh cohort and spectral predictability 0.499. Autocorrelation coming out highest is exactly the invitation to take a second bite, declare it vindicated and quote the number. The pre-registration forbids that in advance: these are descriptive, carry no alpha, and at n=69 sit inside the noise band the earlier permutation null established. No claim is made from them and the shipped diagnostic does not change. Three training-free measures have now been tried on individual serverless workloads and all three have failed, so the diagnostic stays scoped to fleet aggregates rather than being patched. That scope now rests on two independent failures to extend it. Withdraws this project's own recommendation, made two days ago after the literature review, that spectral entropy was the clearest improvement available. The reasoning was sound in the abstract and wrong on this problem, measured twice. The README said it, so the README retracts it. --- README.md | 12 +++++-- docs/EVAL.md | 70 +++++++++++++++++++++++++++++++++++++++ docs/ROADMAP.md | 5 +++ scripts/build_snapshot.py | 9 +++-- 4 files changed, 92 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index a4c1b05..7cbd1d1 100644 --- a/README.md +++ b/README.md @@ -61,8 +61,16 @@ Two consequences are worth stating up front. The incumbent it measures against i Autopilot's percentile recommender, not threshold HPA — beating threshold HPA is routine and proves little. And the predictability diagnostic is a cruder instrument than the field's: spectral entropy is the established forecastability measure, and a single lagged correlation -is known to be weaker, which is very likely why it failed to generalise (see Q13). Replacing -it with spectral entropy is the clearest improvement available to this project. +is known to be weaker. + +That reading suggested an obvious fix, and this project recommended it here. **The fix was +then measured and it does not work.** Spectral entropy lost to the incumbent measure in an +exploratory race and scored AUC 0.499 — chance — on a fresh cohort. A horizon-relative +spectral measure looked promising at AUC 0.700, was pre-registered, and collapsed to 0.526 +on data that had not generated it. Three training-free measures have now been tried on +individual serverless workloads and all three have failed, which is why the diagnostic is +scoped to fleet aggregates rather than patched. See Q13 and Q13b in +[`docs/EVAL.md`](docs/EVAL.md). ## Read the evaluation diff --git a/docs/EVAL.md b/docs/EVAL.md index 44a3383..3a1cec1 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -1300,3 +1300,73 @@ from config, but an apostrophe would have silently returned the wrong SKU's pric unexploitable in production only because of a CDN behaviour nobody had designed for. - **A Pareto summary statistic could rank two frontiers backwards**, and its test passed because the error cancelled between the two curves being compared. + +--- + +# Q13b — the lead did not replicate (2026-08-09) + +Pre-registered in `docs/PREREGISTRATION-Q13B.md` and committed, with the complete analysis +script, **before the confirmatory data existed**. The commit order is checkable in the git +history and is the only thing that makes the rest of this section worth reading. + +## Result + +One measure, one horizon, one test, on 80 workloads drawn fresh and explicitly disjoint from +the 43 the lead was found on. + +| | exploratory (2026-08-08) | confirmatory (2026-08-09) | +|---|---:|---:| +| low-frequency power, AUC @ 12 h | 0.700 | **0.526** | +| p | 0.027 (one of six tests) | **0.359** (one-sided, pre-registered) | +| n measurable | 34 | 69 | + +**Not confirmed.** The effect all but vanished. The design had roughly 0.9 power against a +true AUC of 0.700, so this is not a near miss on an underpowered test — 0.526 is what a +measure with essentially no signal looks like. + +The honest reading is the boring one: **0.700 was the winner's curse.** It was the largest of +six AUCs computed on 34 workloads, and the largest of six noisy estimates is biased upward by +construction. Regression to the mean did the rest. + +## Secondary, and deliberately not acted on + +On the same fresh cohort, daily autocorrelation scored **AUC 0.601** and spectral +predictability **0.499**. + +Autocorrelation scoring highest is exactly the sort of result that invites a second bite: +declare it vindicated, rebuild the diagnostic around it, quote 0.601. That is forbidden here +and the pre-registration says so in advance. These are descriptive numbers carrying no alpha, +computed on n = 69, where the permutation null from the exploratory run put the 95% ceiling +near 0.68 — so 0.601 is inside the noise band and establishes nothing. **No claim is made +from it, and the shipped diagnostic does not change.** + +## What this closes + +- **The low-frequency power measure is refuted**, not merely unconfirmed. It had one + well-powered pre-registered chance and did not take it. +- **Spectral entropy is refuted twice over.** It lost the exploratory race (AUC 0.396 and + 0.456) and scored 0.499 — chance, to three decimals — on fresh data. +- **No training-free measure tested predicts whether forecasting pays on individual + serverless workloads.** Daily autocorrelation, spectral entropy, and horizon-relative + spectral power have all now been tried and all have failed on that population. + +The diagnostic therefore stays exactly where Q13 left it: **valid on fleet-aggregate demand, +where it calls 27 of 28 cells correctly, and scoped out of everything else on the page a +visitor reads.** That scope is now supported by two independent failures to extend it rather +than one. + +## A prediction of ours that was wrong, recorded as such + +After the literature review on 2026-08-08 this project recommended replacing daily +autocorrelation with spectral entropy, calling it "the clearest improvement available". The +forecastability literature does support spectral measures in general, and the reasoning was +sound in the abstract. **On this problem it was wrong**, and measured twice to be wrong. The +recommendation is withdrawn from the README. + +## Added to the negatives ledger + +- **The Q13 replacement measure failed to replicate**, 0.700 → 0.526, against a + pre-registered hypothesis with ~0.9 power. +- **A promising exploratory AUC was the winner's curse**, and pre-registration is what caught + it rather than hindsight. +- **This project publicly recommended an improvement that its own next experiment refuted.** diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index b82ab81..777cef9 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -54,6 +54,11 @@ it measured is narrower and more useful: ignoring the diagnostic entirely — with a non-monotone relationship that no version of the rule predicted. Only the bottom of the range transfers: below ~0.20, forecasting failed to pay on both populations. +5. **Q13b: the replacement does not work either.** Spectral entropy, the field's standard + forecastability measure, scored at chance on a fresh cohort. A horizon-relative spectral + measure reached AUC 0.700 exploratorily, was pre-registered, and fell to 0.526 — the + winner's curse, caught by pre-registration rather than by hindsight. Three measures + tried, three failed, so the diagnostic is scoped rather than patched. The shipped artifact is therefore a **diagnostic that tells you whether to build a forecaster at all**, scoped to aggregated demand and stating that scope on its own results page. That is diff --git a/scripts/build_snapshot.py b/scripts/build_snapshot.py index 72b5df1..b8b0d29 100644 --- a/scripts/build_snapshot.py +++ b/scripts/build_snapshot.py @@ -252,8 +252,13 @@ def findings() -> list[Finding]: "(AUC 0.396 and 0.456). The one candidate showing signal — the share of " "spectral power slower than the commitment window — reaches AUC 0.700 at 12 " "hours but does not survive Bonferroni across the six tests run, so it is " - "recorded as a lead rather than promoted. What survives: below r=0.20, " - "forecasting failed to pay on every population tested." + "recorded as a lead rather than promoted. That lead was then pre-registered " + "and tested on 80 fresh workloads disjoint from the ones that produced it: " + "it fell from AUC 0.700 to 0.526 (one-sided p=0.36) against a design with " + "roughly 0.9 power, so it was the winner's curse and is refuted. Spectral " + "entropy scored 0.499 on the same fresh cohort. Three training-free measures " + "tried, three failed. What survives: below r=0.20, forecasting failed to pay " + "on every population tested." ), ), ]