Answer Q13 and scope the diagnostic to the demand it was measured on - #5
Merged
Conversation
Widened the test from 7 workloads to 43 individual Azure Functions workloads, sampled hostile to the rule — stratified by the diagnostic and oversampled in the borderline band. The 0.50 cutoff scores 51% at a six-hour commitment and 47% at twelve. That is a coin flip, and worse than ignoring the diagnostic and always predicting that forecasting pays. The relationship is not even monotone: win rate climbs out of the noise floor between 0.20 and 0.50 and falls again above 0.70. Two scoring corrections were needed before the number meant anything, and both moved it. Pareto dominance asks whether forecasting is free, not whether it pays: on one workload at r = 0.892 forecasting cut violations 2.7x while costing 6% more and was recorded as a loss. Cells are now scored on total economic cost at the ratio the quantile implies, which is the newsvendor objective already used for Q4. Separately, the median/8 replica heuristic degenerates on serverless traces — a 65%-zeros workload reported 17,413 mean replicas — so sizing is anchored on the p95. What survives is the bottom of the range. Below ~0.20 forecasting failed to pay on both populations, and that is the only part of the diagnostic that transferred. The published 27-of-28 stands and is now stated with its scope: it was measured on fleet aggregates, where summing thousands of VMs or requests leaves daily structure as the dominant exploitable signal. The API verdict, the diagnostic page and the explainer all say so, so a visitor pasting a single spiky function is told the reading is untested for their case rather than given a confident answer the evidence does not support.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Q13 was the one open question: is 0.50 a real boundary, or a round number that seven
workloads flattered? It is closer to the second.
Result
Widened the test to 43 individual Azure Functions workloads — CC-BY, already fetched, no new
licence question — sampled deliberately hostile to the rule: stratified by the diagnostic and
oversampled in the borderline band.
The 0.50 cutoff scores 51% at six hours, 47% at twelve — a coin flip, and worse than the
58%/53% you get by ignoring it and always predicting that forecasting pays. The relationship
is not monotone: it rises out of the noise floor and falls again at the top.
Two scoring corrections, both of which moved the answer
is no worse on both axes — so on
762d22c5a3d7(r = 0.892), cutting violations from0.193 to 0.072 while costing 6% more was recorded as not a win. That asks whether
forecasting is free, not whether it pays. Now scored on total economic cost at
C_u/C_o = q/(1-q), the newsvendor objective already used for Q4.median/8on a 65%-zerosworkload gave a mean of 17,413 replicas. Sizing now anchors on the p95.
dominancealso counts ties as wins. This does not affect the published 27-of-28 —verified: every published win is strict and no plan pair was identical, including
materna-2's 2 of 4 — but it fires often on low-volume functions, so ties are now countedand excluded.
What this changes in the product
The 27-of-28 result stands, now stated with its scope: it was measured on fleet
aggregates, where summing thousands of VMs or requests leaves daily structure as the
dominant exploitable signal. An individual function can carry high daily autocorrelation
while the variance that drives the commitment decision lives inside the window.
The API verdict, the diagnostic page and the explainer now all carry that scope, so a visitor
pasting a single spiky workload is told the reading is untested for their case rather than
handed a confident answer the evidence does not support. Below ~0.20 the diagnostic held on
both populations, and that is stated as the part that transferred.
Gate
ruff clean; mypy strict clean; 181 tests passing;
tsc --noEmitclean and the frontendbuilding.