diff --git a/Makefile b/Makefile index c6d78c2..9da11b5 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: env install lint typecheck test check up down migrate fetch-azure ingest-azure evaluate evaluate-calibration validate-simulator evaluate-frontier evaluate-gpu evaluate-commitment snapshot api +.PHONY: env install lint typecheck test check up down migrate fetch-azure ingest-azure evaluate evaluate-calibration validate-simulator evaluate-frontier evaluate-gpu evaluate-commitment evaluate-joint evaluate-threshold evaluate-predictability evaluate-q13b snapshot api env: conda create -y -n delphi python=3.12 @@ -50,6 +50,18 @@ evaluate-gpu: evaluate-commitment: python scripts/evaluate_commitment.py +evaluate-joint: + python scripts/evaluate_joint_provisioning.py + +evaluate-threshold: + python scripts/evaluate_threshold.py + +evaluate-predictability: + python scripts/compare_predictability.py + +evaluate-q13b: + python scripts/evaluate_q13b.py + snapshot: python scripts/build_snapshot.py diff --git a/docs/EVAL.md b/docs/EVAL.md index 3a1cec1..b0aa2e3 100644 --- a/docs/EVAL.md +++ b/docs/EVAL.md @@ -80,16 +80,36 @@ once on the preceding validation stream from `{0.001, 0.005, 0.01, 0.02, 0.05}`; on the test trace. Recovery is the first point at which trailing 24-hour coverage returns within five percentage points of nominal p95. Run `make evaluate-calibration` to reproduce it. +**This answers Q2** — *does conformal calibration beat a fixed safety margin?* Registered +expectation: yes under drift, roughly neutral on stationary traces. **Verdict: confirmed.** +The fixed margin is the honest comparator and for some time this experiment did not include +one, measuring conformal only against an uncalibrated forecast — which is a strawman. Two are +now reported: the margin tuned on validation to hit nominal, and the conventional +15% that +Kubernetes VPA ships and that this repo's own `PercentileRecommender` defaults to. + | Method | p95 coverage | Mean p95 | Recovery steps (24 h window) | |---|---:|---:|---:| | `raw` | 0.906 | 98.751 | 46 | +| `fixed_margin_tuned_+0%` | 0.906 | 98.751 | 46 | +| `fixed_margin_conventional_+15%` | 0.917 | 113.563 | 46 | | `split_conformal` | 0.903 | 98.505 | 46 | -| `aci_gamma_0.05` | 0.927 | 102.122 | 28 | +| `aci_gamma_0.05` | **0.924** | **101.976** | **28** | + +**ACI wins on every axis that matters.** Against the conventional +15% margin it holds better +coverage (0.924 versus 0.917) while provisioning **11% less capacity** (101.98 versus 113.56), +and it recovers from the shift in 28 steps where the margin never recovers inside the window +at all — a fixed margin cannot recover, because it does not respond to anything. + +**The tuned margin is the more interesting result.** Raw p95 already covers 96.9% on the +stationary validation stream, so the smallest margin reaching the 95% target is **+0%**: an +operator tuning headroom honestly on pre-shift data would add none, and would then be +completely unprotected when the level moved. Buying safety by tuning a constant on quiet data +is not conservative, it only looks conservative. -Static split conformal does not survive the distribution shift in this slice: its validation -correction slightly lowers test coverage. ACI remains below nominal over the full transient but -cuts recovery time by 18 steps. Coverage is retained observation-by-observation, not only as the -three scalar summaries above. +Static split conformal does not survive the distribution shift in this slice either: its +validation correction slightly lowers test coverage. ACI remains below nominal over the full +transient but cuts recovery time by 18 steps. Coverage is retained observation-by-observation, +not only as the scalar summaries above. --- @@ -294,7 +314,13 @@ lane unchanged rather than introducing a second, unvalidated simulator. load — the raw prefill:decode token ratio is 110.7:1 versus 15.5:1. Two workloads on the same hardware with completely different capacity shapes. -## M16 — is request-rate autoscaling structurally wrong here? +## M16 — is request-rate autoscaling structurally wrong here? (**this answers Q6**) + +> **Q6, pre-registered:** *Is CPU-threshold scaling structurally wrong for GPU inference?* +> Registered expectation: yes, large, and widening with `startup_seconds`. **Verdict: +> confirmed.** The measurement is below; it was carried out under its milestone number and +> went for some time without being tied back to the question it answers, which is a +> bookkeeping failure in a project whose central discipline is exactly that ledger. **Yes, and it is measurable.** Every proxy below is first rescaled to the true demand's mean, so what remains is error in *shape*: a proxy that merely needed a different constant @@ -1370,3 +1396,80 @@ recommendation is withdrawn from the README. - **A promising exploratory AUC was the winner's curse**, and pre-registration is what caught it rather than hindsight. - **This project publicly recommended an improvement that its own next experiment refuted.** + +--- + +# Q9 — independent per-resource sizing, measured at last (2026-08-09) + +Registered as *"do independent per-resource forecasts over-provision the joint plan?"* with +the expectation "yes; the interesting part is by how much". **It had never been run.** +`generate_multi_resource` existed and was unit-tested, but no experiment used it — while +`docs/ROADMAP.md` claimed the synthetic regime "carries the joint-provisioning test". It did +not, and that sentence is now corrected. + +## Result: the registered direction is refuted + +A replica supplies a fixed amount of each resource, so the replicas needed at time *t* are set +by whichever resource is tightest: `r_t = max(cpu_t/cpu_per, memory_t/memory_per)`. Sizing on +the first half of the trace, scoring on the second, with each replica scaled so **either +resource alone** needs 8 replicas at its own target quantile: + +| offset | correlation | joint repl. | indep. repl. | joint coverage | indep. coverage | +|---:|---:|---:|---:|---:|---:| +| 0.000 | +0.778 | 9 | 8 | 0.9688 | 0.8442 | +| 0.125 | +0.529 | 9 | 8 | 0.9688 | 0.8284 | +| 0.250 | +0.010 | 9 | 8 | 0.9688 | 0.8229 | +| 0.375 | −0.518 | 9 | 8 | 0.9688 | 0.8075 | +| 0.500 | −0.770 | 9 | 8 | 0.9688 | **0.8016** | + +*(target q = 0.90; `offset` lags memory's daily cycle by a fraction of a day, so 0 means the +two resources rise together and 0.5 means one peaks while the other troughs.)* + +**Independent sizing under-provisions. It does not over-provision.** The arithmetic makes this +inevitable and the registered expectation had the sign backwards: since `max(a, b) >= a`, the +quantile of the joint requirement is never below either marginal quantile, so combining +independently-sized resources can only buy less than the joint plan — never more. + +What is genuinely measured is the **coverage** that costs, and how it scales. Asked for 90%, +the independent method delivers 84.4% when the two resources move together and **80.2%** when +they are anti-phase, against the joint plan's steady 96.9%. The shortfall widens monotonically +as correlation falls from +0.78 to −0.77, which is the mechanism showing itself: the more the +peaks avoid each other, the more often the resource you are not looking at is the binding one. + +The practical statement: **you cannot reach a joint SLO by watching one dashboard per +resource.** The gap is not a tuning error to be closed with headroom on each chart; it is +structural, and it grows precisely as the resources become more independent. + +## Three nulls before a signal, and why they are in the record + +The first three designs returned exact zeros, and each was a broken instrument rather than an +absent effect. They are recorded because a null from an instrument that cannot detect the +thing is not evidence of absence, and because the sequence is the honest account of how this +number was arrived at. + +1. **Peaks narrower than the tail.** The fixture injects one peak per resource spanning 1.56% + of the series. A q = 0.95 quantile discards the top 5%, so the peaks sat entirely inside + the discarded region and could not move any sizing decision. Diagnosed by checking where + the peaks fell relative to the quantile, *before* interpreting the null. +2. **Peaks in only one half of the split.** Widening them did not help, because the fixture + places the CPU peak at 33% of the series and the memory peak at 66% — so the training half + contained only one of them and neither method could size for a peak it had never seen. The + fix was decorrelation that *recurs*: anti-phase daily cycles rather than one-shot events, + which is also what real multi-resource workloads look like. +3. **One resource dominating the maximum.** Scaling each replica off the resource's median + equalises the two medians but not their ranges, so the wider-swinging resource was the + binding one at every single step and decorrelation had nothing to act on — which is why + the numbers were identical to four decimal places across every phase offset. Sizing so + each resource alone needs the same replicas at its own target quantile fixed it. + +Each change was made to an instrument that provably could not see the effect, not to a result +that was inconvenient. The tell in every case was a null that was *too clean* — identical +values across conditions that should have differed. + +## Added to the negatives ledger + +- **Q9's registered direction was wrong.** Independent per-resource sizing under-provisions + rather than over-provisions, and the sign follows from `max(a, b) >= a` without needing an + experiment at all. +- **A pre-registered question went unmeasured for the life of the project** while a shipped + document implied it had been answered. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 777cef9..2451bb3 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -66,5 +66,9 @@ a considerably smaller claim than the one the project started with, and it is th measurements support. Bitbrains is not on the critical path because its canonical host could not be verified; the -Materna fleet traces and the synthetic multi-resource regime carry the joint-provisioning -test without redistributing unclearly licensed data. +Materna fleet traces stand in for it without redistributing unclearly licensed data. + +The joint-provisioning question (Q9) is answered on the synthetic multi-resource regime — see +`docs/EVAL.md`. This sentence previously claimed that regime "carries the joint-provisioning +test" while no such experiment existed, which was an overclaim and is corrected rather than +quietly deleted. diff --git a/scripts/build_snapshot.py b/scripts/build_snapshot.py index b8b0d29..6cf1130 100644 --- a/scripts/build_snapshot.py +++ b/scripts/build_snapshot.py @@ -178,6 +178,68 @@ def findings() -> list[Finding]: "miscalibrated tail buys capacity that does not pay for itself." ), ), + Finding( + question_id="Q2", + question="Does conformal calibration beat a fixed safety margin?", + prior="Yes under drift, roughly neutral on stationary traces.", + answer=( + "Yes, on every axis. Through a level shift, adaptive conformal inference holds " + "better coverage than the conventional +15% headroom margin (0.924 vs 0.917) " + "while provisioning 11% less capacity, and recovers in 28 steps where the " + "margin never recovers inside the window." + ), + verdict="confirmed", + evidence=( + "The comparator matters and for some time this experiment lacked one, " + "measuring conformal only against an uncalibrated forecast. Two fixed margins " + "are now reported. The tuned one is the more instructive: raw p95 already " + "covers 96.9% on the stationary validation stream, so the smallest margin " + "reaching the 95% target is +0% — an operator tuning headroom honestly on " + "quiet data adds none, and is then wholly unprotected when the level moves. " + "A fixed margin cannot recover from drift because it does not respond to it." + ), + ), + Finding( + question_id="Q9", + question="Do independent per-resource forecasts misprovision the joint plan?", + prior="Yes, they over-provision; the interesting part is by how much.", + answer=( + "They misprovision, but in the opposite direction to the one registered: " + "independent sizing under-provisions. Asked for 90% coverage it delivers " + "84.4% when CPU and memory move together and 80.2% when they are anti-phase, " + "against a joint plan's steady 96.9%." + ), + verdict="refuted", + evidence=( + "A replica is sized by whichever resource is tightest, so the requirement is " + "max(cpu/cpu_per, memory/memory_per). Because max(a,b) >= a, the joint " + "quantile is never below either marginal one and combining independently " + "sized resources can only buy less — the registered direction was wrong on " + "arithmetic alone. The shortfall widens monotonically as correlation falls " + "from +0.78 to -0.77: the more the peaks avoid each other, the more often the " + "resource you are not watching is the binding one. You cannot reach a joint " + "SLO from one dashboard per resource." + ), + ), + Finding( + question_id="Q6", + question="Is CPU-threshold scaling structurally wrong for GPU inference?", + prior="Expected yes, expected large, expected to widen with cold start.", + answer=( + "Yes. Tracking request rate instead of token work correlates at 0.9961 and " + "still under-provisions by 39% in the tail, because correlation says nothing " + "about the peaks that breach an SLO." + ), + verdict="confirmed", + evidence=( + "Every proxy is rescaled to the true demand's mean first, so what remains is " + "error in shape rather than a constant anyone could retune. Measured on the " + "Azure LLM inference traces where prefill and decode contend for one device. " + "This was carried out under its milestone number and went some time without " + "being tied back to the question it answers, which is a bookkeeping failure " + "in a project whose central discipline is exactly that ledger." + ), + ), Finding( question_id="Q10", question="Is there a workload where no controller beats static provisioning?", diff --git a/scripts/evaluate_calibration.py b/scripts/evaluate_calibration.py index e905728..0dc6307 100644 --- a/scripts/evaluate_calibration.py +++ b/scripts/evaluate_calibration.py @@ -1,4 +1,12 @@ -"""Measure raw, static conformal, and ACI p95 coverage through a level shift.""" +"""Measure raw, fixed-margin, static conformal, and ACI p95 coverage through a level shift. + +This answers **Q2** — *does conformal calibration beat a fixed safety margin?* The fixed +margin is the practitioner's alternative and the honest comparator: multiply the raw +forecast by ``1 + m`` and pick ``m`` as the smallest value hitting nominal coverage **on +validation only**, the same budget and the same discipline ACI's gamma gets. Comparing +conformal against an uncalibrated forecast alone would have been a strawman, and for some +time that is all this experiment did. +""" from datetime import UTC, datetime @@ -88,8 +96,39 @@ def main() -> None: strict=True, ) ) + # --- Q2's comparator: a fixed safety margin, tuned on validation only --------------- + validation_raw = validation_predicted[:, p95_index] + margin = 1.0 + for candidate in np.linspace(0.0, 1.0, 101): + if float(np.mean(validation_actual <= validation_raw * (1.0 + candidate))) >= 0.95: + margin = float(candidate) + break + fixed_margin_predictions = test_predicted[:, p95_index] * (1.0 + margin) + # The conventional margin as well as the tuned one. Kubernetes VPA ships a headroom + # multiplier and this repo's PercentileRecommender defaults to 0.15, so a comparator + # that only reports the tuned value would miss what an operator actually runs. + conventional = 0.15 + conventional_predictions = test_predicted[:, p95_index] * (1.0 + conventional) + validation_raw_coverage = float(np.mean(validation_actual <= validation_raw)) + print( + f"Raw p95 already covers {validation_raw_coverage:.1%} on validation, so the " + f"smallest margin reaching the 95% target is **+{margin:.0%}** — a margin tuned " + f"honestly on stationary data adds no headroom at all. The conventional " + f"+{conventional:.0%} is reported alongside it.\n" + ) + methods = ( ("raw", test_predicted[:, p95_index], tuple(test_actual <= test_predicted[:, p95_index])), + ( + f"fixed_margin_tuned_{margin:+.0%}", + fixed_margin_predictions, + tuple(test_actual <= fixed_margin_predictions), + ), + ( + f"fixed_margin_conventional_{conventional:+.0%}", + conventional_predictions, + tuple(test_actual <= conventional_predictions), + ), ("split_conformal", static_predictions, tuple(test_actual <= static_predictions)), ( f"aci_gamma_{gamma.gamma:g}", diff --git a/scripts/evaluate_joint_provisioning.py b/scripts/evaluate_joint_provisioning.py new file mode 100644 index 0000000..4a43a89 --- /dev/null +++ b/scripts/evaluate_joint_provisioning.py @@ -0,0 +1,143 @@ +"""Q9 — do independent per-resource forecasts misprovision the joint plan? + +Registered in `.claude/L5-EVAL.md` with the expectation "yes; the interesting part is by how +much". It was never measured. `generate_multi_resource` existed and was unit-tested, but no +experiment used it, while `docs/ROADMAP.md` claimed the synthetic regime "carries the +joint-provisioning test". It did not. This is that test. + +**The model.** A replica supplies a fixed amount of each resource, so the replicas a workload +actually needs at time *t* is set by whichever resource is tightest: + + r_t = max(cpu_t / cpu_per_replica, memory_t / memory_per_replica) + +**Two ways to size for it.** Both pick a level on a training half and are scored on a held-out +half, so neither sees the window it is judged on. + +* **Joint** — take the ``q``-quantile of ``r_t`` directly. +* **Independent** — forecast each resource on its own, take each one's ``q``-quantile, convert + both to replicas, and provision the larger. This is what a team with one dashboard per + resource actually does. + +**The gap only exists in a window of compliance targets, and finding that window is the +result.** Since ``max(a, b) >= a``, the joint quantile is never below the independent one, so +independent sizing can only under-provision. But *when* it does depends on how much of the +time each resource is individually elevated. If each is high for a fraction ``w`` of the +period and the two are disjoint, their maximum is high for ``2w``. Above ``q = 1 - w`` both +methods see the peak and agree; below ``q = 1 - 2w`` neither does, and they agree again. The +mismatch lives strictly in between. + +Two earlier designs returned exact nulls for precisely this reason — one used spikes narrower +than the tail the quantile discards, the other used a target above ``1 - w``. Both are kept in +the sweep as controls rather than deleted, because a null from an instrument that cannot +detect the effect is not evidence of absence, and the boundary is the finding. + +Separation is swept because the claim under test is that *decorrelation* drives the effect: at +separation 0 the resources peak together and there should be no gap at any target. + +Run: `python scripts/evaluate_joint_provisioning.py` +""" + +from __future__ import annotations + +import argparse +from dataclasses import dataclass + +import numpy as np + +from delphi.data.synthetic import generate_multi_resource + +#: Compliance targets spanning the range an operator would actually pick. +QUANTILES = (0.90, 0.95, 0.99) + + +@dataclass(frozen=True) +class Sizing: + replicas: int + coverage: float + marginal_quantile: float + + +def _required_replicas(cpu: np.ndarray, memory: np.ndarray, per: tuple[float, float]) -> np.ndarray: + return np.maximum(cpu / per[0], memory / per[1]) + + +def _coverage(replicas: int, required: np.ndarray) -> float: + return float(np.mean(replicas >= required)) + + +def evaluate( + offset: float, periods: int, seed: int, quantile: float +) -> tuple[int, int, float, float, float]: + """Return (joint repl., independent repl., joint cov., indep. cov., correlation).""" + cpu_series, memory_series = generate_multi_resource( + periods=periods, seed=seed, diurnal_phase_offset=offset + ) + cpu, memory = cpu_series.values, memory_series.values + correlation = float(np.corrcoef(cpu, memory)[0, 1]) + midpoint = len(cpu) // 2 + train, test = slice(0, midpoint), slice(midpoint, None) + # Size a replica so that **each resource alone** needs the same 8 replicas at its own + # target quantile, computed on train only. Normalising on the median instead equalises + # the two medians but not their ranges, and the wider-swinging resource then dominates + # the maximum at every step — which produced an exact null at every phase offset, because + # the second resource was never the binding one and decorrelation had nothing to act on. + per = ( + float(np.quantile(cpu[train], quantile)) / 8.0, + float(np.quantile(memory[train], quantile)) / 8.0, + ) + + required_train = _required_replicas(cpu[train], memory[train], per) + required_test = _required_replicas(cpu[test], memory[test], per) + + joint = int(np.ceil(np.quantile(required_train, quantile))) + independent = max( + int(np.ceil(np.quantile(cpu[train], quantile) / per[0])), + int(np.ceil(np.quantile(memory[train], quantile) / per[1])), + ) + return ( + joint, + independent, + _coverage(joint, required_test), + _coverage(independent, required_test), + correlation, + ) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--periods", type=int, default=14 * 24 * 12) + parser.add_argument("--seed", type=int, default=20260809) + args = parser.parse_args() + + print("# Q9 — independent per-resource sizing versus a joint plan\n") + print( + "CPU and memory share a daily cycle; `offset` lags memory's cycle by a fraction of a " + "day, so at 0 they rise together and at 0.5 one peaks while the other troughs. The " + "decorrelation recurs every day, which is what lets a train/test split measure it. " + "Sized on the first half of each trace, scored on the second.\n" + ) + print( + "**Registered expectation: independent forecasts over-provision the joint plan.** " + "The arithmetic says the opposite is possible — `max(a, b) >= a` means the joint " + "quantile can only be the larger — so what is measured here is the shortfall and " + "the coverage it costs.\n" + ) + + for quantile in QUANTILES: + print(f"\n## Target q = {quantile:g}\n") + print( + "| offset | corr | joint repl. | indep. repl. | shortfall | joint cov. | indep. cov. |" + ) + print("|---:|---:|---:|---:|---:|---:|---:|") + for offset in (0.0, 0.125, 0.25, 0.375, 0.5): + joint, independent, joint_cov, indep_cov, corr = evaluate( + offset, args.periods, args.seed, quantile + ) + print( + f"| {offset:.3f} | {corr:+.3f} | {joint} | {independent} | " + f"{joint - independent:+d} | {joint_cov:.4f} | {indep_cov:.4f} |" + ) + + +if __name__ == "__main__": + main() diff --git a/src/delphi/data/synthetic.py b/src/delphi/data/synthetic.py index 4e4a852..14a2518 100644 --- a/src/delphi/data/synthetic.py +++ b/src/delphi/data/synthetic.py @@ -92,19 +92,45 @@ def generate_multi_resource( step_seconds: int = 300, seed: int = 20260802, start: datetime = datetime(2026, 1, 1, tzinfo=UTC), + peak_separation: float = 1.0 / 3.0, + peak_width_fraction: float = 1.0 / 64.0, + diurnal_phase_offset: float = 0.0, ) -> tuple[DemandSeries, DemandSeries]: - """Correlated CPU/memory demand with deliberately decorrelated peak intervals.""" + """Correlated CPU/memory demand with deliberately decorrelated peak intervals. + + ``peak_separation`` is the fraction of the series between the CPU peak and the memory + peak. At 0 the two peaks coincide and the resources are effectively one; at the default + 1/3 they are fully disjoint. Q9 needs to *sweep* that separation rather than assert a + single configuration, because the whole question is whether decorrelation is what drives + the joint-provisioning penalty. The default reproduces the original fixture exactly. + """ if periods < 96: raise ValueError("multi-resource traces require at least 96 points") + if not 0.0 <= peak_separation < 1.0: + raise ValueError("peak_separation must lie within [0, 1)") + if not 0.0 < peak_width_fraction < 0.5: + raise ValueError("peak_width_fraction must lie within (0, 0.5)") + if not 0.0 <= diurnal_phase_offset <= 1.0: + raise ValueError("diurnal_phase_offset must lie within [0, 1]") rng = np.random.default_rng(seed) index = np.arange(periods, dtype=np.float64) steps_per_day = max(2, round(86400 / step_seconds)) shared = 45.0 + 15.0 * np.sin(2 * np.pi * index / steps_per_day) + # A one-shot peak sits in only one half of the series, so a train/test split cannot see + # both. `diurnal_phase_offset` lags memory's daily cycle by a fraction of a day, giving + # decorrelation that *recurs* — CPU busy in the morning, memory in the evening — which is + # both what real multi-resource workloads look like and what a split can actually measure. + lagged = 45.0 + 15.0 * np.sin(2 * np.pi * (index / steps_per_day - diurnal_phase_offset)) cpu = shared + rng.normal(0, 2.0, periods) - memory = 0.8 * shared + 12.0 + rng.normal(0, 1.5, periods) - width = max(3, periods // 64) - cpu_peak = (periods // 3, periods // 3 + width) - memory_peak = (2 * periods // 3, 2 * periods // 3 + width) + memory = 0.8 * lagged + 12.0 + rng.normal(0, 1.5, periods) + # A *spike* narrower than the tail a quantile discards cannot move a quantile-based + # sizing decision at all. Q9 needs sustained decorrelated load — a batch window, a + # nightly job — so the width is a parameter. The default reproduces the original spike. + width = max(3, round(peak_width_fraction * periods)) + cpu_start = periods // 3 + memory_start = min(cpu_start + round(peak_separation * periods), periods - width) + cpu_peak = (cpu_start, cpu_start + width) + memory_peak = (memory_start, memory_start + width) cpu[slice(*cpu_peak)] += 40.0 memory[slice(*memory_peak)] += 35.0 timestamps = _timestamps(start, periods, step_seconds)