Skip to content

Race three predictability measures; the literature's fix loses - #7

Merged
jadoon200 merged 1 commit into
mainfrom
feat/capacity-control
Aug 8, 2026
Merged

Race three predictability measures; the literature's fix loses#7
jadoon200 merged 1 commit into
mainfrom
feat/capacity-control

Conversation

@jadoon200

Copy link
Copy Markdown
Owner

Q13 said the shipped diagnostic does not generalise. The forecastability literature gave a
reason and a fix — a single lagged correlation reads one frequency, spectral entropy reads
all of them. The fix was tried and it lost.

Result

43 serverless + 5 fleet workloads, same target (did forecasting pay by total economic cost),
threshold-free comparison by AUC, 20,000-draw permutation test:

measure AUC @ 6 h p AUC @ 12 h p
daily autocorrelation (shipped) 0.539 0.67 0.438 0.50
spectral predictability (the field's) 0.396 0.26 0.456 0.63
low-frequency power (ours) 0.595 0.31 0.700 0.027

Spectral entropy failed — at or below a coin flip on serverless, anti-predictive on
fleet. Predicting it would rescue the diagnostic was wrong, and that is recorded.

The only measure with signal is not from the literature. It is the share of spectral
power at periods longer than the commitment window, motivated by this project's own Q1
result: a decision fixed for N hours can only exploit structure slower than N hours. Alone
among the three it is a function of the decision horizon rather than the series alone.

Why the 0.700 is not shipped

Six tests were run. Bonferroni wants p < 0.0083; the permutation p is 0.027. Picking the
largest of six AUCs and quoting its uncorrected p-value is exactly how a result gets
manufactured, and the project's own doctrine says an implausibly clean number is an artefact
until proven otherwise.

Recorded as a lead, not a finding. The follow-up is named: one pre-registered hypothesis,
fresh cohort, horizon fixed in advance. The shipped diagnostic is unchanged and keeps the
scope caveat Q13 forced on it.

Method notes

  • Comparison by AUC, not each measure's own best threshold — fitting a cutoff to 43 points
    would let any measure win. Best-threshold accuracy is printed as a labelled optimistic bound.
  • Numpy only, so the diagnostic keeps no transitive scipy import.
  • The periodogram is Welch-averaged; a raw one is an inconsistent estimator whose entropy is
    dominated by noise.
  • Tests pin the measures against signals with known answers, including two equally-predictable
    series that spectral entropy cannot separate and the window-relative measure can.

Also

Folds the autocorrelation definition into one place. It existed separately in the API and the
evaluation script — the same drift risk that let the shipped verdict contradict docs/EVAL.md
for a day.

Gate

ruff clean; mypy strict clean; 190 tests passing (was 181).

Q13 said the shipped diagnostic does not generalise, and the forecastability
literature gave both a reason and a fix: a single lagged correlation reads one
frequency where spectral entropy reads all of them. The fix was tried against the
same target on the same 43 serverless and 5 fleet workloads. It did not work.

Spectral predictability scores AUC 0.396 at six hours and 0.456 at twelve — at or
below a coin flip, and anti-predictive on the fleet traces. Daily autocorrelation
is no better on serverless (0.539, 0.438). The one measure showing signal is not
from the literature: the share of spectral power at periods longer than the
commitment window, motivated by this project's own Q1 result that a decision
fixed for N hours can only exploit structure slower than N hours. It reaches AUC
0.700 at a twelve-hour commitment.

That 0.700 is not being shipped. Six tests were run, Bonferroni wants p < 0.0083,
and its permutation p is 0.027. Quoting the largest of six AUCs at its uncorrected
p-value is how a result gets manufactured, so it is recorded as a lead and the
follow-up is named: one pre-registered hypothesis, fresh cohort, horizon fixed in
advance. The shipped diagnostic is unchanged and keeps its scope caveat.

Comparison is by AUC rather than each measure's own best threshold, because
fitting a cutoff to 43 points and reporting the accuracy would let any measure
win. Best-threshold accuracy is printed as a labelled optimistic bound.

The measures are numpy-only so the diagnostic keeps no transitive scipy import,
and the periodogram is Welch-averaged because a raw one is inconsistent and its
entropy is dominated by noise. Tests pin them against signals with known answers,
including two equally-predictable series that spectral entropy cannot separate and
the window-relative measure can.

Also folds the autocorrelation definition into one place. It existed separately in
the API and the evaluation script — the same drift risk that let the shipped
verdict contradict docs/EVAL.md for a day.
@jadoon200
jadoon200 merged commit 78fa56f into main Aug 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant