Race three predictability measures; the literature's fix loses - #7
Merged
Conversation
Q13 said the shipped diagnostic does not generalise, and the forecastability literature gave both a reason and a fix: a single lagged correlation reads one frequency where spectral entropy reads all of them. The fix was tried against the same target on the same 43 serverless and 5 fleet workloads. It did not work. Spectral predictability scores AUC 0.396 at six hours and 0.456 at twelve — at or below a coin flip, and anti-predictive on the fleet traces. Daily autocorrelation is no better on serverless (0.539, 0.438). The one measure showing signal is not from the literature: the share of spectral power at periods longer than the commitment window, motivated by this project's own Q1 result that a decision fixed for N hours can only exploit structure slower than N hours. It reaches AUC 0.700 at a twelve-hour commitment. That 0.700 is not being shipped. Six tests were run, Bonferroni wants p < 0.0083, and its permutation p is 0.027. Quoting the largest of six AUCs at its uncorrected p-value is how a result gets manufactured, so it is recorded as a lead and the follow-up is named: one pre-registered hypothesis, fresh cohort, horizon fixed in advance. The shipped diagnostic is unchanged and keeps its scope caveat. Comparison is by AUC rather than each measure's own best threshold, because fitting a cutoff to 43 points and reporting the accuracy would let any measure win. Best-threshold accuracy is printed as a labelled optimistic bound. The measures are numpy-only so the diagnostic keeps no transitive scipy import, and the periodogram is Welch-averaged because a raw one is inconsistent and its entropy is dominated by noise. Tests pin them against signals with known answers, including two equally-predictable series that spectral entropy cannot separate and the window-relative measure can. Also folds the autocorrelation definition into one place. It existed separately in the API and the evaluation script — the same drift risk that let the shipped verdict contradict docs/EVAL.md for a day.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Q13 said the shipped diagnostic does not generalise. The forecastability literature gave a
reason and a fix — a single lagged correlation reads one frequency, spectral entropy reads
all of them. The fix was tried and it lost.
Result
43 serverless + 5 fleet workloads, same target (did forecasting pay by total economic cost),
threshold-free comparison by AUC, 20,000-draw permutation test:
Spectral entropy failed — at or below a coin flip on serverless, anti-predictive on
fleet. Predicting it would rescue the diagnostic was wrong, and that is recorded.
The only measure with signal is not from the literature. It is the share of spectral
power at periods longer than the commitment window, motivated by this project's own Q1
result: a decision fixed for N hours can only exploit structure slower than N hours. Alone
among the three it is a function of the decision horizon rather than the series alone.
Why the 0.700 is not shipped
Six tests were run. Bonferroni wants p < 0.0083; the permutation p is 0.027. Picking the
largest of six AUCs and quoting its uncorrected p-value is exactly how a result gets
manufactured, and the project's own doctrine says an implausibly clean number is an artefact
until proven otherwise.
Recorded as a lead, not a finding. The follow-up is named: one pre-registered hypothesis,
fresh cohort, horizon fixed in advance. The shipped diagnostic is unchanged and keeps the
scope caveat Q13 forced on it.
Method notes
would let any measure win. Best-threshold accuracy is printed as a labelled optimistic bound.
dominated by noise.
series that spectral entropy cannot separate and the window-relative measure can.
Also
Folds the autocorrelation definition into one place. It existed separately in the API and the
evaluation script — the same drift risk that let the shipped verdict contradict
docs/EVAL.mdfor a day.
Gate
ruff clean; mypy strict clean; 190 tests passing (was 181).