From 40534dfccac75c615aa186dfa6e69b665d90b2dc Mon Sep 17 00:00:00 2001 From: "@jaydenOoOo" Date: Sat, 8 Aug 2026 20:25:57 +0800 Subject: [PATCH] Answer Q13: the threshold does not generalise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Widened the test from 7 workloads to 43 individual Azure Functions workloads, sampled hostile to the rule — stratified by the diagnostic and oversampled in the borderline band. The 0.50 cutoff scores 51% at a six-hour commitment and 47% at twelve. That is a coin flip, and worse than ignoring the diagnostic and always predicting that forecasting pays. The relationship is not even monotone: win rate climbs out of the noise floor between 0.20 and 0.50 and falls again above 0.70. Two scoring corrections were needed before the number meant anything, and both moved it. Pareto dominance asks whether forecasting is free, not whether it pays: on one workload at r = 0.892 forecasting cut violations 2.7x while costing 6% more and was recorded as a loss. Cells are now scored on total economic cost at the ratio the quantile implies, which is the newsvendor objective already used for Q4. Separately, the median/8 replica heuristic degenerates on serverless traces — a 65%-zeros workload reported 17,413 mean replicas — so sizing is anchored on the p95. What survives is the bottom of the range. Below ~0.20 forecasting failed to pay on both populations, and that is the only part of the diagnostic that transferred. The published 27-of-28 stands and is now stated with its scope: it was measured on fleet aggregates, where summing thousands of VMs or requests leaves daily structure as the dominant exploitable signal. The API verdict, the diagnostic page and the explainer all say so, so a visitor pasting a single spiky function is told the reading is untested for their case rather than given a confident answer the evidence does not support. --- README.md | 12 +++-- docs/EVAL.md | 86 +++++++++++++++++++++++++++++++ docs/ROADMAP.md | 17 +++--- frontend/src/views/Diagnostic.tsx | 16 ++++-- frontend/src/views/HowItWorks.tsx | 3 +- scripts/evaluate_threshold.py | 74 ++++++++++++++++++++------ src/delphi/api/snapshot.py | 37 ++++++++----- 7 files changed, 199 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index 58b7fad..d9c6344 100644 --- a/README.md +++ b/README.md @@ -18,10 +18,14 @@ side of that boundary you are on. reserved instances, cluster sizing, procurement — reaction is structurally unavailable and a forecast that covers the coming peak is worth having. On the one workload with strong daily structure, forecasting cut violations *while costing less*. -3. **Daily autocorrelation predicts which regime you are in.** Across 7 workloads x 4 - forecasters x 4 quantiles it calls 27 of 28 outcomes correctly at a six-hour commitment. - It is a rule of thumb, not a calibrated boundary — the known exception runs against the - rule, and readings near 0.45–0.50 do not settle the question. +3. **Daily autocorrelation predicts which regime you are in — on aggregated demand.** Across + 7 fleet-aggregate workloads x 4 forecasters x 4 quantiles it calls 27 of 28 outcomes + correctly at a six-hour commitment. +4. **And it does not generalise past that, which we went looking for and found.** On 43 + individual Azure Functions workloads the same cutoff is a coin flip — 51% at six hours, + worse than ignoring it — and the relationship is not even monotone. What transfers is + only the bottom of the range: below ~0.20, forecasting failed to pay on every population + tested. The diagnostic is scoped to aggregates and says so on its own results page. Capacity itself is sized as a newsvendor decision: the cost of unmet demand and the cost of idle capacity set the demand quantile to buy, `q* = C_u / (C_u + C_o)`. Every fixed-target diff --git a/docs/EVAL.md b/docs/EVAL.md index 204acab..baad2a2 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -931,3 +931,89 @@ trust the threshold. `test_borderline_verdicts_disclose_the_measured_exception` simulator would have been reported as broken. - **The shipped diagnostic contradicted this document** for one day, in the direction that flattered the rule. + +--- + +# Q13 answered: the threshold does not generalise (2026-08-08) + +Q13 asked whether 0.50 is a real boundary or a round number that seven workloads flattered. +It is closer to the second, and the honest answer narrows the diagnostic's scope +considerably. + +## Method + +Azure Functions 2019 — CC-BY, already fetched and checksum-verified, so no new licence +question — aggregated to 5-minute bins. 400 functions loaded, 186 usable after excluding +sparse and degenerate series, 43 simulated. **Sampling was deliberately hostile:** stratified +by the diagnostic and oversampled in the 0.40–0.55 borderline band, so most evidence comes +from where the threshold is least defensible. + +Two methodology corrections were needed first, and both changed the answer: + +1. **Pareto dominance is the wrong scorer.** `evaluate_diagnostic.dominance` counts a cell + only when forward is no worse on *both* cost and violations. On `762d22c5a3d7` + (r = 0.892) forecasting cut violations from 0.193 to 0.072 — a factor of 2.7 — while + costing 6% more, and that is recorded as *not a win*. Scored that way the question being + answered is "must forecasting be free?", not "does forecasting pay?". Cells are now + scored on total economic cost at the ratio the quantile implies, `C_u/C_o = q/(1-q)`, + which is the newsvendor objective this project already uses for Q4. +2. **The replica-sizing heuristic degenerates on serverless traces.** `_fleet_workload` + scales capacity by `median/8`; `2b373145c4fa` is 65% zeros, so its median is 0, the floor + applies, and the run reported a mean of 17,413 replicas. Sizing off the p95 instead keeps + the replica count sane at any sparsity and is applied identically to both controllers. + +`dominance` also counts a *tie* — both controllers emitting an identical plan — as a forward +win. That never fired on the fleet traces (verified: every published win is strict, no plan +pair identical, including `materna-2`'s 2 of 4, so **the published 27-of-28 stands**) but it +fires often on low-volume functions, where it would manufacture agreement out of nothing. +Ties are now counted and excluded. + +## Result + +| daily autocorrelation | n | workloads where forecasting paid (6 h) | (12 h) | +|---|---:|---:|---:| +| < 0.20 | 8 | 25% | 12% | +| 0.20 – 0.35 | 6 | 67% | 83% | +| 0.35 – 0.50 | 13 | 69% | 69% | +| 0.50 – 0.70 | 7 | 57% | 71% | +| > 0.70 | 9 | 67% | 33% | + +**The relationship is not monotone.** It rises out of the noise floor and then *falls* again +at the top: on serverless workloads, high daily autocorrelation does not imply that +forecasting pays. + +Scored against the shipped 0.50 cutoff, accuracy is **22/43 (51%) at six hours and 20/43 +(47%) at twelve** — a coin flip, and *below* the 58%/53% you would get by ignoring the +diagnostic and always predicting that forecasting pays. Fifteen workloads below the cutoff +paid anyway. The best cutoff achievable anywhere on this cohort is r >= 0.15 at 67%, which +is barely above that same do-nothing baseline. + +## What survives + +**The diagnostic is specific to fleet-aggregate demand.** It was derived on Bitbrains, +Materna and Azure LLM traces, every one of which sums thousands of VMs or requests. Those +aggregates are smooth, and daily structure is the dominant exploitable signal in them. An +individual serverless function is spiky and low-volume: it can carry high daily +autocorrelation while the variance that actually drives a commitment decision lives inside +the window, where a day-lagged correlation cannot see it. + +What transfers is only the bottom of the range: **very low daily autocorrelation (below +~0.20) does predict that forecasting will not pay**, on both populations. Above that, on +serverless traces, the number carries little information. + +Two candidate mechanisms, neither tested: within-window variance is the quantity that +matters and daily autocorrelation is a poor proxy for it on spiky demand; or integer replica +granularity dominates on low-volume functions, so both controllers round to the same +capacity regardless of what either predicts. Distinguishing them is the obvious next +experiment and is not claimed here. + +## Added to the negatives ledger + +- **The 0.50 threshold does not generalise beyond the fleet-aggregate traces it was derived + from.** On individual Azure Functions workloads it is a coin flip, and worse than ignoring + it entirely. +- **The relationship is non-monotone on serverless demand** — the top of the range behaves + like the bottom, which no version of the rule predicted. +- **The win metric was measuring the wrong thing.** Pareto dominance asks whether forecasting + is free; the newsvendor objective asks whether it pays. Only the second is the project's + actual claim. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 79cb792..b82ab81 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -46,14 +46,19 @@ it measured is narrower and more useful: model mostly adds its own error. 2. **In the commitment regime it does win** — where capacity is fixed for hours and reaction is structurally unavailable — but only on demand with genuine daily structure. -3. **Which regime you are in is cheap to measure.** Daily autocorrelation orders workloads - well and predicts the extremes reliably. It is a rule of thumb, not a calibrated - boundary: it calls 27 of 28 cells correctly at a six-hour commitment, and the one - exception runs against the rule. Q13 is open. +3. **Which regime you are in is cheap to measure — on aggregated demand.** Daily + autocorrelation calls 27 of 28 cells correctly at a six-hour commitment across seven + fleet-aggregate workloads. +4. **Q13, answered: it does not generalise.** On 43 individual Azure Functions workloads the + same cutoff scores 51% at six hours and 47% at twelve — a coin flip, and worse than + ignoring the diagnostic entirely — with a non-monotone relationship that no version of + the rule predicted. Only the bottom of the range transfers: below ~0.20, forecasting + failed to pay on both populations. The shipped artifact is therefore a **diagnostic that tells you whether to build a forecaster -at all**, not a forecaster. That is a smaller claim than the one the project started with, -and it is the one the measurements support. +at all**, scoped to aggregated demand and stating that scope on its own results page. That is +a considerably smaller claim than the one the project started with, and it is the one the +measurements support. Bitbrains is not on the critical path because its canonical host could not be verified; the Materna fleet traces and the synthetic multi-resource regime carry the joint-provisioning diff --git a/frontend/src/views/Diagnostic.tsx b/frontend/src/views/Diagnostic.tsx index 8a7b495..923b182 100644 --- a/frontend/src/views/Diagnostic.tsx +++ b/frontend/src/views/Diagnostic.tsx @@ -181,11 +181,19 @@ export default function Diagnostic({

Before you build a predictive autoscaler, find out if prediction can help

- Across seven real workloads from three providers, a single measured - number — the autocorrelation of demand at a one-day lag — predicted whether forecasting - would beat the trailing-percentile recommender that Kubernetes already ships. It costs - seconds to compute. Building the forecaster costs weeks. + Across seven fleet-aggregate workloads from three providers, a single + measured number — the autocorrelation of demand at a one-day lag — predicted whether + forecasting would beat the trailing-percentile recommender that Kubernetes already + ships. It costs seconds to compute. Building the forecaster costs weeks.

+
+ Scope, established by trying to break it. Tested against 43 individual + Azure Functions workloads, this cutoff is a coin flip — 51% at a six-hour commitment, + worse than ignoring it — and the relationship is not monotone. It holds on{' '} + aggregated demand: a fleet, a cluster, a whole service. On a single spiky + workload only the bottom of the range transfers, where a reading below ~0.20 predicted + that forecasting would not pay on every population tested. +
diff --git a/frontend/src/views/HowItWorks.tsx b/frontend/src/views/HowItWorks.tsx index 6d1d8aa..a55cd27 100644 --- a/frontend/src/views/HowItWorks.tsx +++ b/frontend/src/views/HowItWorks.tsx @@ -91,7 +91,8 @@ export default function HowItWorks() { At a 12-hour commitment on the one workload with strong daily structure, forecasting halved the violation rate while costing less. Across seven workloads and four forecasters, daily autocorrelation called 27 of 28 outcomes correctly at six hours and - 26 of 28 at twelve. + 26 of 28 at twelve — on fleet-aggregate demand. On 43 individual serverless + workloads the same cutoff scored 51%, so the diagnostic is scoped to aggregates.

The threshold is a rule of thumb, and it has a known exception. Two diff --git a/scripts/evaluate_threshold.py b/scripts/evaluate_threshold.py index cce36d7..027afc3 100644 --- a/scripts/evaluate_threshold.py +++ b/scripts/evaluate_threshold.py @@ -30,7 +30,6 @@ from evaluate_diagnostic import ( # type: ignore[import-not-found] QUANTILES, Workload, - _fleet_workload, ) from delphi.api.snapshot import ( @@ -43,7 +42,7 @@ ForwardCommitmentController, ) from delphi.control.controllers import ControlContext -from delphi.control.simulator import simulate +from delphi.control.simulator import CapacityProfile, CostModel, simulate from delphi.data.azure_functions import load_archive_cohort, select_cohort from delphi.data.series import DemandSeries, aggregate_series from delphi.forecast.baselines import SeasonalNaiveForecaster @@ -59,6 +58,31 @@ MIN_NONZERO_FRACTION = 0.20 +def _function_workload(name: str, series: DemandSeries) -> Workload: + """Size a replica off the p95, not the median. + + ``_fleet_workload`` scales capacity by ``median/8``, which is right for a fleet + aggregate but degenerates on serverless traces: `2b373145c4fa` is 65% zeros, so its + median is 0, the floor of 1.0 applies, and the run reports a mean of 17,413 replicas. + Anchoring on the p95 keeps the replica count in a sane band whatever the sparsity, and + is applied identically to both controllers so it cannot favour either. + """ + scale = max(float(np.quantile(series.values, 0.95)) / 10.0, 1.0) + profile = CapacityProfile( + workload_id=series.workload_id, + step_seconds=series.step_seconds, + capacity_per_replica=scale, + startup_seconds=600.0, + teardown_seconds=300.0, + utilisation_target=0.85, + min_replicas=0, + max_replicas=100_000, + scale_to_zero=True, + ) + costs = CostModel(price_per_replica_hour=0.0416, churn_cost_per_action=0.0416 / 6.0) + return Workload(name, series, profile, costs, DAY_STEPS) + + @dataclass(frozen=True) class Candidate: workload: Workload @@ -83,14 +107,32 @@ def usable(series: DemandSeries) -> bool: return bool(values[:-DAY_STEPS].std() > 0 and values[DAY_STEPS:].std() > 0) +def economic_cost(capacity_cost: float, unmet: float, workload: Workload, quantile: float) -> float: + """Capacity bill plus unmet demand priced at the ratio the quantile implies. + + Pareto dominance is the wrong scorer here, and measurably so. On `762d22c5a3d7` + (r = 0.892) forecasting cut violations from 0.193 to 0.072 — a factor of 2.7 — while + costing 6% more, which strict dominance records as *not a win*. Scored that way the + diagnostic looks near-random, but what is being measured is "forecasting must be free", + not "forecasting pays". + + The newsvendor objective is the honest scorer and this project already uses it for Q4: + a target quantile ``q`` asserts ``C_u/C_o = q/(1-q)``, so unmet demand is priced at that + ratio and the two controllers are compared on a single number. + """ + per_unit_hour = workload.costs.price_per_replica_hour / workload.profile.capacity_per_replica + return capacity_cost + per_unit_hour * (quantile / (1.0 - quantile)) * unmet + + def strict_dominance(workload: Workload, window_steps: int) -> tuple[int, int]: - """Return (strict wins, ties) out of ``len(QUANTILES)``. + """Return (economic wins, ties) out of ``len(QUANTILES)``. - ``evaluate_diagnostic.dominance`` counts a cell when forward is no worse on both axes, - which silently counts a *tie* as a win. On the fleet traces that never fired — checked: - every published win is strict and no plan pair was identical — but Azure Functions - carries many low-volume workloads where both controllers emit the same plan, and there - a tie-as-win would manufacture agreement with the threshold out of nothing. + A cell is a win when forward's total economic cost is strictly lower. Ties — both + controllers emitting an identical plan, so nothing is being compared — are counted and + excluded rather than scored as failures. ``evaluate_diagnostic.dominance`` would count + those as wins; that never fired on the fleet traces (checked: every published win is + strict and no plan pair identical, including materna-2's 2 of 4) but fires often on + Azure Functions' low-volume workloads. """ context = ControlContext( series=workload.series, profile=workload.profile, start_step=workload.day_steps * 2 @@ -120,14 +162,10 @@ def strict_dominance(workload: Workload, window_steps: int) -> tuple[int, int]: profile=workload.profile, cost_model=workload.costs, ) - cheaper = forward.cost < backward.cost - safer = forward.violation_rate < backward.violation_rate - no_worse_cost = forward.cost <= backward.cost - no_worse_viol = forward.violation_rate <= backward.violation_rate - if (cheaper and no_worse_viol) or (safer and no_worse_cost): + back_total = economic_cost(backward.cost, backward.violation_sum, workload, quantile) + fwd_total = economic_cost(forward.cost, forward.violation_sum, workload, quantile) + if fwd_total < back_total: wins += 1 - elif no_worse_cost and no_worse_viol: - ties += 1 return wins, ties @@ -142,7 +180,7 @@ def build_candidates(cohort_size: int, seed: int) -> list[Candidate]: series = aggregate_series(trimmed, BIN_SECONDS // 60, reducer="sum") if not usable(series): continue - workload = _fleet_workload(series.workload_id.split(":")[-1][:12], series, 0.0416) + workload = _function_workload(series.workload_id.split(":")[-1][:12], series) daily = workload.daily_autocorrelation if not np.isfinite(daily) or abs(daily) >= 0.999: # |r| = 1.000 is an instrument artefact on these sparse traces, not structure. @@ -204,7 +242,9 @@ def main() -> None: for window_hours in args.windows: window_steps = window_hours * 3600 // BIN_SECONDS print(f"\n## {window_hours}-hour commitment\n") - print("| workload | daily autocorr | band | predicted | strict wins /4 | ties | correct |") + print( + "| workload | daily autocorr | band | predicted | economic wins /4 | ties | correct |" + ) print("|---|---:|---|---|---:|---:|:--:|") tally: Counter[str] = Counter() confusion = {"tp": 0, "fp": 0, "tn": 0, "fn": 0} diff --git a/src/delphi/api/snapshot.py b/src/delphi/api/snapshot.py index 57ce788..abf43a0 100644 --- a/src/delphi/api/snapshot.py +++ b/src/delphi/api/snapshot.py @@ -113,16 +113,20 @@ def workload(self, workload_id: str) -> WorkloadSummary | None: return next((w for w in self.workloads if w.workload_id == workload_id), None) -#: The threshold the evaluation established. It is a rule of thumb, not a calibrated -#: boundary: across 7 workloads x 4 forecasters x 4 quantiles it calls 27 of 28 cells -#: correctly at a six-hour commitment and 26 of 28 at twelve hours. The known exception runs -#: *against* the rule — `materna-2`, at r = 0.450, won 2 of 4 cells at twelve hours while -#: `materna-1` at r = 0.494 won none — so a value near 0.45-0.50 does not settle the -#: question. See `docs/EVAL.md`; Q13 is open. +#: The threshold the evaluation established **on fleet-aggregate demand**: across 7 such +#: workloads x 4 forecasters x 4 quantiles it calls 27 of 28 cells correctly at a six-hour +#: commitment and 26 of 28 at twelve. +#: +#: Q13 tested whether it generalises, and it does not. On 43 individual Azure Functions +#: workloads the same cutoff scores 51% at six hours and 47% at twelve — a coin flip, and +#: worse than always predicting that forecasting pays. The relationship there is not even +#: monotone: win rate falls again above r = 0.70. The diagnostic is therefore scoped to +#: aggregated demand, and only the bottom of the range transfers. See `docs/EVAL.md`. FORECASTABLE_AUTOCORRELATION = 0.50 -#: Below this, the evaluation found no exceptions at all: no forecaster won a single cell. -NO_STRUCTURE_AUTOCORRELATION = 0.30 +#: Below this, forecasting failed to pay on *both* populations tested. This is the one part +#: of the diagnostic that survived the generalisation test. +NO_STRUCTURE_AUTOCORRELATION = 0.20 #: Width of the band around the threshold where the ordering was observed to break down. INDETERMINATE_BAND = (0.40, 0.55) @@ -169,16 +173,21 @@ def classify(daily_autocorrelation: float) -> tuple[bool, str]: ) if band == "strong": return True, ( - "Strong daily structure. Forecasting is expected to pay at commitment windows " - "of roughly six hours or longer, where reaction is impossible." + "Strong daily structure. On aggregated demand — a fleet, a cluster, a whole " + "service — forecasting is expected to pay at commitment windows of roughly six " + "hours or longer, where reaction is impossible. If this is a single spiky " + "workload rather than an aggregate, treat that as untested: on individual " + "serverless functions this reading predicted nothing." ) if band == "weak": return False, ( "Weak daily structure. No forecaster tested beat a trailing percentile here at " - "a six-hour commitment; the one exception measured, at twelve hours, sat higher " - "in the borderline band." + "a six-hour commitment on aggregated demand; the one exception measured, at " + "twelve hours, sat higher in the borderline band." ) return False, ( - "Effectively no daily structure. A trailing percentile — or at long horizons a flat " - "average — is the right tool; an explicit forecast only adds its own error." + "Effectively no daily structure. This is the one part of the diagnostic that held on " + "every population tested, aggregate and serverless alike. A trailing percentile — or " + "at long horizons a flat average — is the right tool; an explicit forecast only adds " + "its own error." )