From 40016c60f9928e887b085725baf17f34ce6f0172 Mon Sep 17 00:00:00 2001 From: "@jaydenOoOo" Date: Sun, 9 Aug 2026 19:34:46 +0800 Subject: [PATCH] Make the two docs a reviewer opens next tell the truth MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ARCHITECTURE.md still described a sixth layer of specialist agents exchanging typed proposals under a supervisor, with a decision ledger. That layer was cut weeks ago and does not exist. A reviewer opening the architecture document was being shown a multi-agent system the repository does not contain — the same class of overclaim already fixed three times in the served surface, surviving in the one document nobody had re-read. Rewritten to describe the five layers that exist, with the cut stated explicitly rather than silently deleted. DATA.md omitted Materna entirely. Materna supplies three of the seven workloads in the diagnostic study and the materna-2 exception that keeps Q13 honest, and its code has carried LICENCE = "unverified-host-unreachable" the whole time. The doctrine is that an unverified licence is a stop condition rather than permission to omit the record, and the record was omitted. Bitbrains had a one-line mention with no detail. Both now have a full entry: the canonical Grid Workloads Archive host is unreachable, the traces come from the at-large mirror, checksums are committed, nothing is redistributed, and five of seven workloads depend on them. The distinction that justifies using them at all — depending on a checksummed fetch is weaker exposure than redistributing bytes — is stated rather than assumed, and Alibaba is contrasted as the case that was dropped instead. Also sets the repository description, homepage and topics on GitHub. The description still advertised the original thesis rather than what the project became, and the live URL appeared only in the README, not in the About panel a reviewer reads first. --- docs/ARCHITECTURE.md | 84 ++++++++++++++++++++++++++++++++++++-------- docs/DATA.md | 32 +++++++++++++++-- 2 files changed, 99 insertions(+), 17 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 5bfe5b9..c25fe10 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,18 +1,72 @@ # Architecture -DELPHI is layered so forecasting quality and decision quality can fail independently and be -measured independently. - -1. **Data** normalises every source into provenance-stamped, UTC-explicit demand points plus a - workload profile that declares serving capacity and actuation delay. -2. **Forecast** emits quantiles over a horizon with an empirical coverage record. -3. **Control** selects a quantile from the under/over-provision cost ratio and evaluates the - resulting plan in a validated replay simulator. -4. **Specialists** own forecast, sizing, cost, SLO, drift, carbon, and verification concerns. - They communicate through typed proposals; deterministic code performs arbitration. -5. **Product** exposes the capacity picture, decision ledger, assumptions, and regret for human - review. No real infrastructure is actuated. - -The database spine currently implements layer 1. Later schemas are introduced only with the -milestone that proves their behaviour. +DELPHI is layered so that **forecasting quality and decision quality can fail independently +and be measured independently**. That separation is the point: the project's central result +is that a better forecast does not produce a better decision, which is only observable if the +two are not entangled in one component. +```text + public trace / synthetic demand + │ + ┌───────▼────────┐ + │ 1. Data │ provenance-stamped, UTC-explicit demand points + a workload + └───────┬────────┘ profile declaring serving capacity and actuation delay + │ + ┌───────▼────────┐ + │ 2. Diagnostic │ is this workload forecastable at all? training-free, seconds + └───────┬────────┘ to compute — answered before anything is built + │ + ┌───────▼────────┐ + │ 3. Forecast │ quantiles over a horizon, with an empirical coverage record; + └───────┬────────┘ conformal / ACI calibration measured on held-out time + │ + ┌───────▼────────┐ + │ 4. Control │ q* = C_u/(C_u+C_o) selects the quantile from the price of + └───────┬────────┘ failure; the plan is scored in a validated replay simulator + │ + ┌───────▼────────┐ + │ 5. Product │ read-only API + dashboard: capacity picture, findings, + └────────────────┘ assumptions and limits. No real infrastructure is actuated. +``` + +## Layer notes + +**1. Data.** Every source is normalised into the same `DemandSeries` — explicit UTC +timestamps, per-point imputation flags and quality, and a declared licence. Missing points +are represented, never forward-filled silently. Sources and their licence status are recorded +in [`DATA.md`](DATA.md), including the ones whose terms could not be verified. + +**2. Diagnostic.** Daily autocorrelation, computed in seconds without training anything. It +is deliberately the first thing the product does, because the most useful answer this project +found is often *"do not build a forecaster for this workload"*. Its scope is narrow and +stated: it holds on fleet-aggregate demand and does not transfer to individual serverless +workloads — see Q13 and Q13b in [`EVAL.md`](EVAL.md). + +**3. Forecast.** Forecasters emit quantiles, never a bare point estimate, because the layer +above consumes a quantile. Coverage is measured as a time series rather than assumed, and no +model's native quantile head is trusted without an empirical calibration layer over it. Five +model families are implemented; none of them changes the capacity decision. + +**4. Control.** The compliance target is *derived* rather than chosen: a controller pinned at +p95 is asserting that unmet demand costs 19x idle capacity, and this layer states that ratio +instead of inheriting it from convention. Plans are scored in a replay simulator validated +against the Erlang-C closed form, with an explicit actuation delay — the parameter that +decides whether forecasting can pay at all. + +**5. Product.** Read-only. Every response that shows a comparison carries its assumptions as +structured data, so a client cannot render the numbers without the caveats. Nothing here +actuates a real cluster, and the deployed snapshot declares itself a demo rather than +inferring liveness from freshness. + +## What is deliberately absent + +An earlier revision of this document described a sixth layer of **specialist agents** +exchanging typed proposals under a supervisor, with a decision ledger. **That layer was cut +and does not exist.** Its own pre-registered expectation (Q5) was that it would not improve +decision quality, only auditability, so cutting it removed a likely-null result rather than a +likely finding. Nothing in the product claims a ledger, and this document should not have +gone on describing one. The cut and its reasoning are recorded in [`ROADMAP.md`](ROADMAP.md). + +The database spine implements layer 1. Later layers hold their state in baked snapshots +rather than in schemas, because every experiment here takes minutes to hours and none of it +can run inside a web request. diff --git a/docs/DATA.md b/docs/DATA.md index 2f9c303..fcec13b 100644 --- a/docs/DATA.md +++ b/docs/DATA.md @@ -7,8 +7,9 @@ cohort rules, and small test fixtures are committed; raw data remains under igno |---|---|---|---| | Deterministic synthetic gold set | MIT, generated locally | configurable | M1 deterministic and negative-control tests | | Azure Functions 2019 | CC-BY 4.0; citation required | 1 minute | Downloaded 2026-08-03; 142,968,140 bytes; SHA-256 `aff8b3ca7240a41a109e4ee598e0a96e45fcb92e7b8395ac19cb3748cd260d89` | -| Azure LLM/LMM inference | CC-BY 4.0 | request-level | planned inference lane | -| Bitbrains GWA-T-12 | terms not currently verifiable; canonical host unavailable | 5 minutes | optional multi-resource trace | +| Azure LLM/LMM inference | CC-BY 4.0 | request-level | **GPU lane (M14–M16)**; 44M requests, code + conv | +| Bitbrains GWA-T-12 | **terms not verifiable — canonical host unreachable** | 5 minutes | two fleets in the diagnostic study | +| Materna GWA-T-13 | **terms not verifiable — canonical host unreachable** | 5 minutes | three fleets in the diagnostic study, incl. the `materna-2` exception | | Alibaba cluster traces | no explicit licence found as of 2026-08-03; data will not be redistributed | event/hourly | gated, optional extension | The Azure evaluation cohort is selected from day 1 using the 20 highest-volume functions plus @@ -20,6 +21,33 @@ Retrieval date, byte size, SHA-256, exact citation, and cohort-selection seed ar source is fetched or selected. An unverified licence is a stop condition, not permission to omit the record. +## Bitbrains GWA-T-12 and Materna GWA-T-13 — used, with the licence unresolved + +Both carry `LICENCE = "unverified-host-unreachable"` in code, and both appear in published +results, so the record belongs here rather than only in a source file. + +| Field | Value | +|---|---| +| Canonical host | `gwa.ewi.tudelft.nl` — the Grid Workloads Archive. **Connection refused, verified twice on 2026-08-02.** | +| Obtained from | `atlarge-research.com/gwa-traces/` mirror | +| Materna SHA-256 | `1380879f0de17cb57619e55c312b41f26ef95743a382f41692db72168fd9afb4` | +| Licence status | **Not verifiable.** The archive's terms page is served by the unreachable canonical host; the mirror publishes no licence file. | +| Citation | Recorded in `src/delphi/data/materna.py` and `src/delphi/data/bitbrains.py` | + +**Neither is redistributed.** Fetch scripts pull from the mirror at run time, checksums are +committed, and `data/` is gitignored — so the repository contains no bytes of either trace. + +**What this means for the results.** Five of the seven workloads in the diagnostic study are +Bitbrains or Materna fleets, including `materna-2`, which supplies the exception that keeps +Q13 honest. Those results are reproducible by anyone who can reach the mirror, and they are +reported with this caveat attached rather than quietly. Had the project depended on +*redistributing* the data, an unverified licence would have been a stop condition; depending +on a checksummed fetch of a publicly mirrored academic trace is a weaker exposure, and the +distinction is stated rather than assumed. + +The Alibaba traces were treated more strictly and dropped entirely, because that lane would +have required a scale extension rather than a reproducibility path. + ## Azure LLM inference traces 2024 | Field | Value |