diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 5bfe5b9..c25fe10 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,18 +1,72 @@ # Architecture -DELPHI is layered so forecasting quality and decision quality can fail independently and be -measured independently. - -1. **Data** normalises every source into provenance-stamped, UTC-explicit demand points plus a - workload profile that declares serving capacity and actuation delay. -2. **Forecast** emits quantiles over a horizon with an empirical coverage record. -3. **Control** selects a quantile from the under/over-provision cost ratio and evaluates the - resulting plan in a validated replay simulator. -4. **Specialists** own forecast, sizing, cost, SLO, drift, carbon, and verification concerns. - They communicate through typed proposals; deterministic code performs arbitration. -5. **Product** exposes the capacity picture, decision ledger, assumptions, and regret for human - review. No real infrastructure is actuated. - -The database spine currently implements layer 1. Later schemas are introduced only with the -milestone that proves their behaviour. +DELPHI is layered so that **forecasting quality and decision quality can fail independently +and be measured independently**. That separation is the point: the project's central result +is that a better forecast does not produce a better decision, which is only observable if the +two are not entangled in one component. +```text + public trace / synthetic demand + │ + ┌───────▼────────┐ + │ 1. Data │ provenance-stamped, UTC-explicit demand points + a workload + └───────┬────────┘ profile declaring serving capacity and actuation delay + │ + ┌───────▼────────┐ + │ 2. Diagnostic │ is this workload forecastable at all? training-free, seconds + └───────┬────────┘ to compute — answered before anything is built + │ + ┌───────▼────────┐ + │ 3. Forecast │ quantiles over a horizon, with an empirical coverage record; + └───────┬────────┘ conformal / ACI calibration measured on held-out time + │ + ┌───────▼────────┐ + │ 4. Control │ q* = C_u/(C_u+C_o) selects the quantile from the price of + └───────┬────────┘ failure; the plan is scored in a validated replay simulator + │ + ┌───────▼────────┐ + │ 5. Product │ read-only API + dashboard: capacity picture, findings, + └────────────────┘ assumptions and limits. No real infrastructure is actuated. +``` + +## Layer notes + +**1. Data.** Every source is normalised into the same `DemandSeries` — explicit UTC +timestamps, per-point imputation flags and quality, and a declared licence. Missing points +are represented, never forward-filled silently. Sources and their licence status are recorded +in [`DATA.md`](DATA.md), including the ones whose terms could not be verified. + +**2. Diagnostic.** Daily autocorrelation, computed in seconds without training anything. It +is deliberately the first thing the product does, because the most useful answer this project +found is often *"do not build a forecaster for this workload"*. Its scope is narrow and +stated: it holds on fleet-aggregate demand and does not transfer to individual serverless +workloads — see Q13 and Q13b in [`EVAL.md`](EVAL.md). + +**3. Forecast.** Forecasters emit quantiles, never a bare point estimate, because the layer +above consumes a quantile. Coverage is measured as a time series rather than assumed, and no +model's native quantile head is trusted without an empirical calibration layer over it. Five +model families are implemented; none of them changes the capacity decision. + +**4. Control.** The compliance target is *derived* rather than chosen: a controller pinned at +p95 is asserting that unmet demand costs 19x idle capacity, and this layer states that ratio +instead of inheriting it from convention. Plans are scored in a replay simulator validated +against the Erlang-C closed form, with an explicit actuation delay — the parameter that +decides whether forecasting can pay at all. + +**5. Product.** Read-only. Every response that shows a comparison carries its assumptions as +structured data, so a client cannot render the numbers without the caveats. Nothing here +actuates a real cluster, and the deployed snapshot declares itself a demo rather than +inferring liveness from freshness. + +## What is deliberately absent + +An earlier revision of this document described a sixth layer of **specialist agents** +exchanging typed proposals under a supervisor, with a decision ledger. **That layer was cut +and does not exist.** Its own pre-registered expectation (Q5) was that it would not improve +decision quality, only auditability, so cutting it removed a likely-null result rather than a +likely finding. Nothing in the product claims a ledger, and this document should not have +gone on describing one. The cut and its reasoning are recorded in [`ROADMAP.md`](ROADMAP.md). + +The database spine implements layer 1. Later layers hold their state in baked snapshots +rather than in schemas, because every experiment here takes minutes to hours and none of it +can run inside a web request. diff --git a/docs/DATA.md b/docs/DATA.md index 2f9c303..fcec13b 100644 --- a/docs/DATA.md +++ b/docs/DATA.md @@ -7,8 +7,9 @@ cohort rules, and small test fixtures are committed; raw data remains under igno |---|---|---|---| | Deterministic synthetic gold set | MIT, generated locally | configurable | M1 deterministic and negative-control tests | | Azure Functions 2019 | CC-BY 4.0; citation required | 1 minute | Downloaded 2026-08-03; 142,968,140 bytes; SHA-256 `aff8b3ca7240a41a109e4ee598e0a96e45fcb92e7b8395ac19cb3748cd260d89` | -| Azure LLM/LMM inference | CC-BY 4.0 | request-level | planned inference lane | -| Bitbrains GWA-T-12 | terms not currently verifiable; canonical host unavailable | 5 minutes | optional multi-resource trace | +| Azure LLM/LMM inference | CC-BY 4.0 | request-level | **GPU lane (M14–M16)**; 44M requests, code + conv | +| Bitbrains GWA-T-12 | **terms not verifiable — canonical host unreachable** | 5 minutes | two fleets in the diagnostic study | +| Materna GWA-T-13 | **terms not verifiable — canonical host unreachable** | 5 minutes | three fleets in the diagnostic study, incl. the `materna-2` exception | | Alibaba cluster traces | no explicit licence found as of 2026-08-03; data will not be redistributed | event/hourly | gated, optional extension | The Azure evaluation cohort is selected from day 1 using the 20 highest-volume functions plus @@ -20,6 +21,33 @@ Retrieval date, byte size, SHA-256, exact citation, and cohort-selection seed ar source is fetched or selected. An unverified licence is a stop condition, not permission to omit the record. +## Bitbrains GWA-T-12 and Materna GWA-T-13 — used, with the licence unresolved + +Both carry `LICENCE = "unverified-host-unreachable"` in code, and both appear in published +results, so the record belongs here rather than only in a source file. + +| Field | Value | +|---|---| +| Canonical host | `gwa.ewi.tudelft.nl` — the Grid Workloads Archive. **Connection refused, verified twice on 2026-08-02.** | +| Obtained from | `atlarge-research.com/gwa-traces/` mirror | +| Materna SHA-256 | `1380879f0de17cb57619e55c312b41f26ef95743a382f41692db72168fd9afb4` | +| Licence status | **Not verifiable.** The archive's terms page is served by the unreachable canonical host; the mirror publishes no licence file. | +| Citation | Recorded in `src/delphi/data/materna.py` and `src/delphi/data/bitbrains.py` | + +**Neither is redistributed.** Fetch scripts pull from the mirror at run time, checksums are +committed, and `data/` is gitignored — so the repository contains no bytes of either trace. + +**What this means for the results.** Five of the seven workloads in the diagnostic study are +Bitbrains or Materna fleets, including `materna-2`, which supplies the exception that keeps +Q13 honest. Those results are reproducible by anyone who can reach the mirror, and they are +reported with this caveat attached rather than quietly. Had the project depended on +*redistributing* the data, an unverified licence would have been a stop condition; depending +on a checksummed fetch of a publicly mirrored academic trace is a weaker exposure, and the +distinction is stated rather than assumed. + +The Alibaba traces were treated more strictly and dropped entirely, because that lane would +have required a scale extension rather than a reproducibility path. + ## Azure LLM inference traces 2024 | Field | Value |