Check the conclusions against published work - #6
Merged
Conversation
A finding nobody else has seen is usually an instrument artefact, so every headline claim was checked against the literature. Six are corroborated, one is a re-derivation that should be cited, one needed narrowing, and none was contradicted. Re-ran evaluate_gpu_frontier.py, which produced the project's central negative result and which the earlier correction pass missed: every controller in it re-plans each step, so all of them were affected by the actuation-delay bug. The verdict is unchanged at 0 of 8, and the widening cost gap is now cleanly monotone where it previously turned over at the last step. Numbers updated. The thesis that a controller consumes a decision rather than a point estimate is decision-focused learning, an established framework this project reached independently via calibration and cites nowhere. Recorded as prior art rather than presented as new. The diagnostic is a weaker instrument than the field's. Spectral entropy is the established forecastability measure and recent work proposes spectral predictability for exactly this job — deciding without training whether forecasting will beat a simple baseline. Its stated advantage over a single lagged correlation is precisely the failure Q13 measured, so the Q13 negative is what that literature predicts. BACC already orders its traces by autocorrelation and spectral entropy together. Narrowed the LightGBM claim. Pre-training does improve cloud-workload forecast accuracy, so "sophistication does not help" is wrong; what the evidence supports is that a more accurate forecaster did not produce better decisions at these horizons. LightGBM is also only the strongest model in this repo, and the foundation models were cut, so the question is open rather than settled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A finding nobody else has seen is usually an instrument artefact, so every headline claim was
checked against the literature. Six corroborated, one is a re-derivation that should be
cited, one needed narrowing, none contradicted.
Caught a gap in the previous correction pass
evaluate_gpu_frontier.pyproduced the project's central negative result — forecasting neverbeats the incumbent percentile recommender — and every controller in it re-plans at each
step, so all of them were affected by the actuation-delay bug. The earlier pass re-ran the
frontier and GPU lane but missed this one.
Re-run: verdict unchanged at 0 of 8. The widening cost gap is now cleanly monotone on
conv(12.0% → 17.5% → 18.8% → 23.7%) where the original turned over at the last step(27.2% → 24.1%). The central negative result is robust to the bug.
Corroborated
CR = C_u/(C_u+C_o)is textbook and cloud provisioning is a namedapplication. Prior art on the non-stationary case with predictions exists and is closer to
this project than anything in
RESEARCH.md.OOM-hit jobs. Measuring against it rather than threshold HPA is why the negative is
interesting: published proactive wins are usually against threshold HPA, where DELPHI wins
too.
both match current practitioner guidance.
Re-derivation to cite, not claim
The premise that a controller consumes a decision rather than a point estimate is the
founding position of decision-focused learning / predict-then-optimize, whose established
result is that better accuracy does not in general give better decisions. The project reached
it independently via calibration and cites neither term. Sound finding, standard framing —
presenting it as novel would misread the field.
The diagnostic is a weaker instrument than the field's
Spectral entropy is the established forecastability measure, and recent work proposes
spectral predictability for exactly our job: deciding without training whether forecasting
beats a simple baseline. Its stated advantage over a single lagged correlation is precisely
the failure Q13 measured — one lag versus structure across all frequencies. So the Q13
negative is what that literature predicts, which is the strongest evidence yet that the
measurement apparatus is working. BACC already orders its traces by autocorrelation and
spectral entropy.
Narrowed
Pre-training does improve cloud-workload forecast accuracy (CloudOps benchmark: 27% error
reduction), so "sophistication does not help" is wrong. What the evidence supports is that a
more accurate forecaster did not produce better decisions at these horizons. LightGBM is
also only the strongest model in this repo, and M17's foundation models were cut — so the
question is open, not settled.
Gate
ruff clean; mypy strict clean; 181 tests passing.