Answer Q13 and scope the diagnostic to the demand it was measured on

README.md

@@ -18,10 +18,14 @@ side of that boundary you are on.
    reserved instances, cluster sizing, procurement — reaction is structurally unavailable
    and a forecast that covers the coming peak is worth having. On the one workload with
    strong daily structure, forecasting cut violations *while costing less*.
-3. **Daily autocorrelation predicts which regime you are in.** Across 7 workloads x 4
-   forecasters x 4 quantiles it calls 27 of 28 outcomes correctly at a six-hour commitment.
-   It is a rule of thumb, not a calibrated boundary — the known exception runs against the
-   rule, and readings near 0.45–0.50 do not settle the question.
+3. **Daily autocorrelation predicts which regime you are in — on aggregated demand.** Across
+   7 fleet-aggregate workloads x 4 forecasters x 4 quantiles it calls 27 of 28 outcomes
+   correctly at a six-hour commitment.
+4. **And it does not generalise past that, which we went looking for and found.** On 43
+   individual Azure Functions workloads the same cutoff is a coin flip — 51% at six hours,
+   worse than ignoring it — and the relationship is not even monotone. What transfers is
+   only the bottom of the range: below ~0.20, forecasting failed to pay on every population
+   tested. The diagnostic is scoped to aggregates and says so on its own results page.
 
 Capacity itself is sized as a newsvendor decision: the cost of unmet demand and the cost of
 idle capacity set the demand quantile to buy, `q* = C_u / (C_u + C_o)`. Every fixed-target

docs/EVAL.md

@@ -931,3 +931,89 @@ trust the threshold. `test_borderline_verdicts_disclose_the_measured_exception`
   simulator would have been reported as broken.
 - **The shipped diagnostic contradicted this document** for one day, in the direction that
   flattered the rule.
+
+---
+
+# Q13 answered: the threshold does not generalise (2026-08-08)
+
+Q13 asked whether 0.50 is a real boundary or a round number that seven workloads flattered.
+It is closer to the second, and the honest answer narrows the diagnostic's scope
+considerably.
+
+## Method
+
+Azure Functions 2019 — CC-BY, already fetched and checksum-verified, so no new licence
+question — aggregated to 5-minute bins. 400 functions loaded, 186 usable after excluding
+sparse and degenerate series, 43 simulated. **Sampling was deliberately hostile:** stratified
+by the diagnostic and oversampled in the 0.40–0.55 borderline band, so most evidence comes
+from where the threshold is least defensible.
+
+Two methodology corrections were needed first, and both changed the answer:
+
+1. **Pareto dominance is the wrong scorer.** `evaluate_diagnostic.dominance` counts a cell
+   only when forward is no worse on *both* cost and violations. On `762d22c5a3d7`
+   (r = 0.892) forecasting cut violations from 0.193 to 0.072 — a factor of 2.7 — while
+   costing 6% more, and that is recorded as *not a win*. Scored that way the question being
+   answered is "must forecasting be free?", not "does forecasting pay?". Cells are now
+   scored on total economic cost at the ratio the quantile implies, `C_u/C_o = q/(1-q)`,
+   which is the newsvendor objective this project already uses for Q4.
+2. **The replica-sizing heuristic degenerates on serverless traces.** `_fleet_workload`
+   scales capacity by `median/8`; `2b373145c4fa` is 65% zeros, so its median is 0, the floor
+   applies, and the run reported a mean of 17,413 replicas. Sizing off the p95 instead keeps
+   the replica count sane at any sparsity and is applied identically to both controllers.
+
+`dominance` also counts a *tie* — both controllers emitting an identical plan — as a forward
+win. That never fired on the fleet traces (verified: every published win is strict, no plan
+pair identical, including `materna-2`'s 2 of 4, so **the published 27-of-28 stands**) but it
+fires often on low-volume functions, where it would manufacture agreement out of nothing.
+Ties are now counted and excluded.
+
+## Result
+
+| daily autocorrelation | n | workloads where forecasting paid (6 h) | (12 h) |
+|---|---:|---:|---:|
+| < 0.20 | 8 | 25% | 12% |
+| 0.20 – 0.35 | 6 | 67% | 83% |
+| 0.35 – 0.50 | 13 | 69% | 69% |
+| 0.50 – 0.70 | 7 | 57% | 71% |
+| > 0.70 | 9 | 67% | 33% |
+
+**The relationship is not monotone.** It rises out of the noise floor and then *falls* again
+at the top: on serverless workloads, high daily autocorrelation does not imply that
+forecasting pays.
+
+Scored against the shipped 0.50 cutoff, accuracy is **22/43 (51%) at six hours and 20/43
+(47%) at twelve** — a coin flip, and *below* the 58%/53% you would get by ignoring the
+diagnostic and always predicting that forecasting pays. Fifteen workloads below the cutoff
+paid anyway. The best cutoff achievable anywhere on this cohort is r >= 0.15 at 67%, which
+is barely above that same do-nothing baseline.
+
+## What survives
+
+**The diagnostic is specific to fleet-aggregate demand.** It was derived on Bitbrains,
+Materna and Azure LLM traces, every one of which sums thousands of VMs or requests. Those
+aggregates are smooth, and daily structure is the dominant exploitable signal in them. An
+individual serverless function is spiky and low-volume: it can carry high daily
+autocorrelation while the variance that actually drives a commitment decision lives inside
+the window, where a day-lagged correlation cannot see it.
+
+What transfers is only the bottom of the range: **very low daily autocorrelation (below
+~0.20) does predict that forecasting will not pay**, on both populations. Above that, on
+serverless traces, the number carries little information.
+
+Two candidate mechanisms, neither tested: within-window variance is the quantity that
+matters and daily autocorrelation is a poor proxy for it on spiky demand; or integer replica
+granularity dominates on low-volume functions, so both controllers round to the same
+capacity regardless of what either predicts. Distinguishing them is the obvious next
+experiment and is not claimed here.
+
+## Added to the negatives ledger
+
+- **The 0.50 threshold does not generalise beyond the fleet-aggregate traces it was derived
+  from.** On individual Azure Functions workloads it is a coin flip, and worse than ignoring
+  it entirely.
+- **The relationship is non-monotone on serverless demand** — the top of the range behaves
+  like the bottom, which no version of the rule predicted.
+- **The win metric was measuring the wrong thing.** Pareto dominance asks whether forecasting
+  is free; the newsvendor objective asks whether it pays. Only the second is the project's
+  actual claim.

docs/ROADMAP.md

@@ -46,14 +46,19 @@ it measured is narrower and more useful:
    model mostly adds its own error.
 2. **In the commitment regime it does win** — where capacity is fixed for hours and reaction
    is structurally unavailable — but only on demand with genuine daily structure.
-3. **Which regime you are in is cheap to measure.** Daily autocorrelation orders workloads
-   well and predicts the extremes reliably. It is a rule of thumb, not a calibrated
-   boundary: it calls 27 of 28 cells correctly at a six-hour commitment, and the one
-   exception runs against the rule. Q13 is open.
+3. **Which regime you are in is cheap to measure — on aggregated demand.** Daily
+   autocorrelation calls 27 of 28 cells correctly at a six-hour commitment across seven
+   fleet-aggregate workloads.
+4. **Q13, answered: it does not generalise.** On 43 individual Azure Functions workloads the
+   same cutoff scores 51% at six hours and 47% at twelve — a coin flip, and worse than
+   ignoring the diagnostic entirely — with a non-monotone relationship that no version of
+   the rule predicted. Only the bottom of the range transfers: below ~0.20, forecasting
+   failed to pay on both populations.
 
 The shipped artifact is therefore a **diagnostic that tells you whether to build a forecaster
-at all**, not a forecaster. That is a smaller claim than the one the project started with,
-and it is the one the measurements support.
+at all**, scoped to aggregated demand and stating that scope on its own results page. That is
+a considerably smaller claim than the one the project started with, and it is the one the
+measurements support.
 
 Bitbrains is not on the critical path because its canonical host could not be verified; the
 Materna fleet traces and the synthetic multi-resource regime carry the joint-provisioning

frontend/src/views/Diagnostic.tsx

@@ -181,11 +181,19 @@ export default function Diagnostic({
       <section className="panel">
         <h2>Before you build a predictive autoscaler, find out if prediction can help</h2>
         <p className="sub">
-          Across <strong>seven real workloads from three providers</strong>, a single measured
-          number — the autocorrelation of demand at a one-day lag — predicted whether forecasting
-          would beat the trailing-percentile recommender that Kubernetes already ships. It costs
-          seconds to compute. Building the forecaster costs weeks.
+          Across <strong>seven fleet-aggregate workloads from three providers</strong>, a single
+          measured number — the autocorrelation of demand at a one-day lag — predicted whether
+          forecasting would beat the trailing-percentile recommender that Kubernetes already
+          ships. It costs seconds to compute. Building the forecaster costs weeks.
         </p>
+        <div className="callout limits">
+          <strong>Scope, established by trying to break it.</strong> Tested against 43 individual
+          Azure Functions workloads, this cutoff is a coin flip — 51% at a six-hour commitment,
+          worse than ignoring it — and the relationship is not monotone. It holds on{' '}
+          <em>aggregated</em> demand: a fleet, a cluster, a whole service. On a single spiky
+          workload only the bottom of the range transfers, where a reading below ~0.20 predicted
+          that forecasting would not pay on every population tested.
+        </div>
 
         <div className="grid two">
           <div>

frontend/src/views/HowItWorks.tsx

@@ -91,7 +91,8 @@ export default function HowItWorks() {
           At a 12-hour commitment on the one workload with strong daily structure, forecasting
           halved the violation rate <em>while costing less</em>. Across seven workloads and four
           forecasters, daily autocorrelation called 27 of 28 outcomes correctly at six hours and
-          26 of 28 at twelve.
+          26 of 28 at twelve — on <em>fleet-aggregate</em> demand. On 43 individual serverless
+          workloads the same cutoff scored 51%, so the diagnostic is scoped to aggregates.
         </p>
         <div className="callout limits" style={{ marginTop: 14 }}>
           <strong>The threshold is a rule of thumb, and it has a known exception.</strong> Two

scripts/evaluate_threshold.py

@@ -30,7 +30,6 @@
 from evaluate_diagnostic import (  # type: ignore[import-not-found]
     QUANTILES,
     Workload,
-    _fleet_workload,
 )
 
 from delphi.api.snapshot import (
@@ -43,7 +42,7 @@
     ForwardCommitmentController,
 )
 from delphi.control.controllers import ControlContext
-from delphi.control.simulator import simulate
+from delphi.control.simulator import CapacityProfile, CostModel, simulate
 from delphi.data.azure_functions import load_archive_cohort, select_cohort
 from delphi.data.series import DemandSeries, aggregate_series
 from delphi.forecast.baselines import SeasonalNaiveForecaster
@@ -59,6 +58,31 @@
 MIN_NONZERO_FRACTION = 0.20
 
 
+def _function_workload(name: str, series: DemandSeries) -> Workload:
+    """Size a replica off the p95, not the median.
+
+    ``_fleet_workload`` scales capacity by ``median/8``, which is right for a fleet
+    aggregate but degenerates on serverless traces: `2b373145c4fa` is 65% zeros, so its
+    median is 0, the floor of 1.0 applies, and the run reports a mean of 17,413 replicas.
+    Anchoring on the p95 keeps the replica count in a sane band whatever the sparsity, and
+    is applied identically to both controllers so it cannot favour either.
+    """
+    scale = max(float(np.quantile(series.values, 0.95)) / 10.0, 1.0)
+    profile = CapacityProfile(
+        workload_id=series.workload_id,
+        step_seconds=series.step_seconds,
+        capacity_per_replica=scale,
+        startup_seconds=600.0,
+        teardown_seconds=300.0,
+        utilisation_target=0.85,
+        min_replicas=0,
+        max_replicas=100_000,
+        scale_to_zero=True,
+    )
+    costs = CostModel(price_per_replica_hour=0.0416, churn_cost_per_action=0.0416 / 6.0)
+    return Workload(name, series, profile, costs, DAY_STEPS)
+
+
 @dataclass(frozen=True)
 class Candidate:
     workload: Workload
@@ -83,14 +107,32 @@ def usable(series: DemandSeries) -> bool:
     return bool(values[:-DAY_STEPS].std() > 0 and values[DAY_STEPS:].std() > 0)
 
 
+def economic_cost(capacity_cost: float, unmet: float, workload: Workload, quantile: float) -> float:
+    """Capacity bill plus unmet demand priced at the ratio the quantile implies.
+
+    Pareto dominance is the wrong scorer here, and measurably so. On `762d22c5a3d7`
+    (r = 0.892) forecasting cut violations from 0.193 to 0.072 — a factor of 2.7 — while
+    costing 6% more, which strict dominance records as *not a win*. Scored that way the
+    diagnostic looks near-random, but what is being measured is "forecasting must be free",
+    not "forecasting pays".
+
+    The newsvendor objective is the honest scorer and this project already uses it for Q4:
+    a target quantile ``q`` asserts ``C_u/C_o = q/(1-q)``, so unmet demand is priced at that
+    ratio and the two controllers are compared on a single number.
+    """
+    per_unit_hour = workload.costs.price_per_replica_hour / workload.profile.capacity_per_replica
+    return capacity_cost + per_unit_hour * (quantile / (1.0 - quantile)) * unmet
+
+
 def strict_dominance(workload: Workload, window_steps: int) -> tuple[int, int]:
-    """Return (strict wins, ties) out of ``len(QUANTILES)``.
+    """Return (economic wins, ties) out of ``len(QUANTILES)``.
 
-    ``evaluate_diagnostic.dominance`` counts a cell when forward is no worse on both axes,
-    which silently counts a *tie* as a win. On the fleet traces that never fired — checked:
-    every published win is strict and no plan pair was identical — but Azure Functions
-    carries many low-volume workloads where both controllers emit the same plan, and there
-    a tie-as-win would manufacture agreement with the threshold out of nothing.
+    A cell is a win when forward's total economic cost is strictly lower. Ties — both
+    controllers emitting an identical plan, so nothing is being compared — are counted and
+    excluded rather than scored as failures. ``evaluate_diagnostic.dominance`` would count
+    those as wins; that never fired on the fleet traces (checked: every published win is
+    strict and no plan pair identical, including materna-2's 2 of 4) but fires often on
+    Azure Functions' low-volume workloads.
     """
     context = ControlContext(
         series=workload.series, profile=workload.profile, start_step=workload.day_steps * 2
@@ -120,14 +162,10 @@ def strict_dominance(workload: Workload, window_steps: int) -> tuple[int, int]:
             profile=workload.profile,
             cost_model=workload.costs,
         )
-        cheaper = forward.cost < backward.cost
-        safer = forward.violation_rate < backward.violation_rate
-        no_worse_cost = forward.cost <= backward.cost
-        no_worse_viol = forward.violation_rate <= backward.violation_rate
-        if (cheaper and no_worse_viol) or (safer and no_worse_cost):
+        back_total = economic_cost(backward.cost, backward.violation_sum, workload, quantile)
+        fwd_total = economic_cost(forward.cost, forward.violation_sum, workload, quantile)
+        if fwd_total < back_total:
             wins += 1
-        elif no_worse_cost and no_worse_viol:
-            ties += 1
     return wins, ties
 
 
@@ -142,7 +180,7 @@ def build_candidates(cohort_size: int, seed: int) -> list[Candidate]:
         series = aggregate_series(trimmed, BIN_SECONDS // 60, reducer="sum")
         if not usable(series):
             continue
-        workload = _fleet_workload(series.workload_id.split(":")[-1][:12], series, 0.0416)
+        workload = _function_workload(series.workload_id.split(":")[-1][:12], series)
         daily = workload.daily_autocorrelation
         if not np.isfinite(daily) or abs(daily) >= 0.999:
             # |r| = 1.000 is an instrument artefact on these sparse traces, not structure.
@@ -204,7 +242,9 @@ def main() -> None:
     for window_hours in args.windows:
         window_steps = window_hours * 3600 // BIN_SECONDS
         print(f"\n## {window_hours}-hour commitment\n")
-        print("| workload | daily autocorr | band | predicted | strict wins /4 | ties | correct |")
+        print(
+            "| workload | daily autocorr | band | predicted | economic wins /4 | ties | correct |"
+        )
         print("|---|---:|---|---|---:|---:|:--:|")
         tally: Counter[str] = Counter()
         confusion = {"tp": 0, "fp": 0, "tn": 0, "fn": 0}