Skip to content

Make the two docs a reviewer opens next tell the truth #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jadoon200 merged 1 commit into from
Aug 9, 2026
Merged

Make the two docs a reviewer opens next tell the truth #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 69 additions & 15 deletions docs/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,72 @@
# Architecture

DELPHI is layered so forecasting quality and decision quality can fail independently and be
measured independently.

1. **Data** normalises every source into provenance-stamped, UTC-explicit demand points plus a
workload profile that declares serving capacity and actuation delay.
2. **Forecast** emits quantiles over a horizon with an empirical coverage record.
3. **Control** selects a quantile from the under/over-provision cost ratio and evaluates the
resulting plan in a validated replay simulator.
4. **Specialists** own forecast, sizing, cost, SLO, drift, carbon, and verification concerns.
They communicate through typed proposals; deterministic code performs arbitration.
5. **Product** exposes the capacity picture, decision ledger, assumptions, and regret for human
review. No real infrastructure is actuated.

The database spine currently implements layer 1. Later schemas are introduced only with the
milestone that proves their behaviour.
DELPHI is layered so that **forecasting quality and decision quality can fail independently
and be measured independently**. That separation is the point: the project's central result
is that a better forecast does not produce a better decision, which is only observable if the
two are not entangled in one component.

```text
public trace / synthetic demand
│
┌───────▼────────┐
│ 1. Data │ provenance-stamped, UTC-explicit demand points + a workload
└───────┬────────┘ profile declaring serving capacity and actuation delay
│
┌───────▼────────┐
│ 2. Diagnostic │ is this workload forecastable at all? training-free, seconds
└───────┬────────┘ to compute — answered before anything is built
│
┌───────▼────────┐
│ 3. Forecast │ quantiles over a horizon, with an empirical coverage record;
└───────┬────────┘ conformal / ACI calibration measured on held-out time
│
┌───────▼────────┐
│ 4. Control │ q* = C_u/(C_u+C_o) selects the quantile from the price of
└───────┬────────┘ failure; the plan is scored in a validated replay simulator
│
┌───────▼────────┐
│ 5. Product │ read-only API + dashboard: capacity picture, findings,
└────────────────┘ assumptions and limits. No real infrastructure is actuated.
```

## Layer notes

**1. Data.** Every source is normalised into the same `DemandSeries` — explicit UTC
timestamps, per-point imputation flags and quality, and a declared licence. Missing points
are represented, never forward-filled silently. Sources and their licence status are recorded
in [`DATA.md`](DATA.md), including the ones whose terms could not be verified.

**2. Diagnostic.** Daily autocorrelation, computed in seconds without training anything. It
is deliberately the first thing the product does, because the most useful answer this project
found is often *"do not build a forecaster for this workload"*. Its scope is narrow and
stated: it holds on fleet-aggregate demand and does not transfer to individual serverless
workloads — see Q13 and Q13b in [`EVAL.md`](EVAL.md).

**3. Forecast.** Forecasters emit quantiles, never a bare point estimate, because the layer
above consumes a quantile. Coverage is measured as a time series rather than assumed, and no
model's native quantile head is trusted without an empirical calibration layer over it. Five
model families are implemented; none of them changes the capacity decision.

**4. Control.** The compliance target is *derived* rather than chosen: a controller pinned at
p95 is asserting that unmet demand costs 19x idle capacity, and this layer states that ratio
instead of inheriting it from convention. Plans are scored in a replay simulator validated
against the Erlang-C closed form, with an explicit actuation delay — the parameter that
decides whether forecasting can pay at all.

**5. Product.** Read-only. Every response that shows a comparison carries its assumptions as
structured data, so a client cannot render the numbers without the caveats. Nothing here
actuates a real cluster, and the deployed snapshot declares itself a demo rather than
inferring liveness from freshness.

## What is deliberately absent

An earlier revision of this document described a sixth layer of **specialist agents**
exchanging typed proposals under a supervisor, with a decision ledger. **That layer was cut
and does not exist.** Its own pre-registered expectation (Q5) was that it would not improve
decision quality, only auditability, so cutting it removed a likely-null result rather than a
likely finding. Nothing in the product claims a ledger, and this document should not have
gone on describing one. The cut and its reasoning are recorded in [`ROADMAP.md`](ROADMAP.md).

The database spine implements layer 1. Later layers hold their state in baked snapshots
rather than in schemas, because every experiment here takes minutes to hours and none of it
can run inside a web request.
32 changes: 30 additions & 2 deletions docs/DATA.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ cohort rules, and small test fixtures are committed; raw data remains under igno
|---|---|---|---|
| Deterministic synthetic gold set | MIT, generated locally | configurable | M1 deterministic and negative-control tests |
| Azure Functions 2019 | CC-BY 4.0; citation required | 1 minute | Downloaded 2026-08-03; 142,968,140 bytes; SHA-256 `aff8b3ca7240a41a109e4ee598e0a96e45fcb92e7b8395ac19cb3748cd260d89` |
| Azure LLM/LMM inference | CC-BY 4.0 | request-level | planned inference lane |
| Bitbrains GWA-T-12 | terms not currently verifiable; canonical host unavailable | 5 minutes | optional multi-resource trace |
| Azure LLM/LMM inference | CC-BY 4.0 | request-level | **GPU lane (M14–M16)**; 44M requests, code + conv |
| Bitbrains GWA-T-12 | **terms not verifiable — canonical host unreachable** | 5 minutes | two fleets in the diagnostic study |
| Materna GWA-T-13 | **terms not verifiable — canonical host unreachable** | 5 minutes | three fleets in the diagnostic study, incl. the `materna-2` exception |
| Alibaba cluster traces | no explicit licence found as of 2026-08-03; data will not be redistributed | event/hourly | gated, optional extension |

The Azure evaluation cohort is selected from day 1 using the 20 highest-volume functions plus
Expand All @@ -20,6 +21,33 @@ Retrieval date, byte size, SHA-256, exact citation, and cohort-selection seed ar
source is fetched or selected. An unverified licence is a stop condition, not permission to omit
the record.

## Bitbrains GWA-T-12 and Materna GWA-T-13 — used, with the licence unresolved

Both carry `LICENCE = "unverified-host-unreachable"` in code, and both appear in published
results, so the record belongs here rather than only in a source file.

| Field | Value |
|---|---|
| Canonical host | `gwa.ewi.tudelft.nl` — the Grid Workloads Archive. **Connection refused, verified twice on 2026-08-02.** |
| Obtained from | `atlarge-research.com/gwa-traces/` mirror |
| Materna SHA-256 | `1380879f0de17cb57619e55c312b41f26ef95743a382f41692db72168fd9afb4` |
| Licence status | **Not verifiable.** The archive's terms page is served by the unreachable canonical host; the mirror publishes no licence file. |
| Citation | Recorded in `src/delphi/data/materna.py` and `src/delphi/data/bitbrains.py` |

**Neither is redistributed.** Fetch scripts pull from the mirror at run time, checksums are
committed, and `data/` is gitignored — so the repository contains no bytes of either trace.

**What this means for the results.** Five of the seven workloads in the diagnostic study are
Bitbrains or Materna fleets, including `materna-2`, which supplies the exception that keeps
Q13 honest. Those results are reproducible by anyone who can reach the mirror, and they are
reported with this caveat attached rather than quietly. Had the project depended on
*redistributing* the data, an unverified licence would have been a stop condition; depending
on a checksummed fetch of a publicly mirrored academic trace is a weaker exposure, and the
distinction is stated rather than assumed.

The Alibaba traces were treated more strictly and dropped entirely, because that lane would
have required a scale extension rather than a reproducibility path.

## Azure LLM inference traces 2024

| Field | Value |
Expand Down
Loading