Q13b: the pre-registered lead did not replicate - #10
Merged
Conversation
The low-frequency power lead reached AUC 0.700 at a twelve-hour commitment with permutation p = 0.027, which does not survive Bonferroni across the six tests that produced it. A lead becomes a finding by surviving one pre-specified test on data that did not generate it, so this commits the hypothesis and the complete analysis before the confirmatory data exists. The commit order is the only evidence that claim is worth anything. One measure, one horizon, one test. Twelve hours only — testing both windows would restore the multiplicity problem the lead already failed. Alpha 0.05, one-sided because the hypothesis predicts a direction, 20,000 permutations. Target n = 80, chosen for roughly 0.9 power against the AUC the exploratory run suggested; a null at this size means the effect is smaller than we thought, not that it is absent. The cohort is drawn fresh under a new seed and explicitly excludes the 43 workloads the lead was found on, regenerated deterministically rather than hard-coded. Sampling is uniform with no stratification: the exploratory run oversampled the borderline band, and stratifying on the predictor under test would bias the statistic being tested. The decision rule is written down with no third option. No re-slicing by band, and no falling back to the six-hour window if twelve disappoints. Daily autocorrelation and spectral predictability are computed on the same cohort as descriptive comparisons that carry no alpha and cannot confirm anything.
Ran the test pre-registered in the previous commit. The horizon-relative spectral measure fell from AUC 0.700 to 0.526 on 80 fresh workloads disjoint from the ones that produced it, one-sided p = 0.36 against a design with roughly 0.9 power. Not a near miss on an underpowered test — 0.526 is what no signal looks like. The honest reading is the boring one. 0.700 was the largest of six AUCs computed on 34 workloads, and the largest of six noisy estimates is biased upward by construction. Pre-registration is what caught that rather than hindsight, which is the entire reason it was written first. Daily autocorrelation scored 0.601 on the same fresh cohort and spectral predictability 0.499. Autocorrelation coming out highest is exactly the invitation to take a second bite, declare it vindicated and quote the number. The pre-registration forbids that in advance: these are descriptive, carry no alpha, and at n=69 sit inside the noise band the earlier permutation null established. No claim is made from them and the shipped diagnostic does not change. Three training-free measures have now been tried on individual serverless workloads and all three have failed, so the diagnostic stays scoped to fleet aggregates rather than being patched. That scope now rests on two independent failures to extend it. Withdraws this project's own recommendation, made two days ago after the literature review, that spectral entropy was the clearest improvement available. The reasoning was sound in the abstract and wrong on this problem, measured twice. The README said it, so the README retracts it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Q13 replacement measure was pre-registered in the previous commit — hypothesis, horizon,
sample size, decision rule and full analysis script, committed and pushed before the
confirmatory data existed. This is the result.
It did not replicate
80 workloads drawn fresh under a new seed and explicitly disjoint from the 43 that produced
the lead. The design had ~0.9 power against a true AUC of 0.700, so this is not a near miss
on an underpowered test.
0.700 was the winner's curse — the largest of six AUCs on 34 workloads, and the largest
of six noisy estimates is biased upward by construction. Pre-registration caught it rather
than hindsight, which is exactly what it was written first to do.
The secondary result I am not acting on
On the same fresh cohort, daily autocorrelation scored 0.601 and spectral predictability
0.499.
Autocorrelation coming out highest is precisely the invitation to take a second bite: declare
it vindicated, rebuild around it, quote 0.601. The pre-registration forbids that in advance.
These carry no alpha, and at n = 69 they sit inside the noise band the earlier permutation
null established (95% ceiling ≈ 0.68). No claim is made from them.
What this closes
pre-registered chance, not taken.
scored 0.499 — chance — on fresh data.
serverless workloads. Three tried, three failed.
The diagnostic stays exactly where Q13 left it: valid on fleet-aggregate demand at 27 of 28
cells, and scoped out of everything else on the page a visitor reads. That scope now rests on
two independent failures to extend it rather than one.
A recommendation of ours, retracted
Two days ago, after the literature review, this project recommended replacing daily
autocorrelation with spectral entropy and called it "the clearest improvement available". The
reasoning was sound in the abstract; on this problem it was wrong, and now measured twice to
be wrong. The README said it, so the README retracts it.
The shipped Q13 finding carries the null result too, so the live site states it.
Gate
ruff clean; mypy strict clean; 201 tests — all verified by exit code.