Skip to content

Close M17, and make EVAL.md readable in order - #12

Merged
jadoon200 merged 2 commits into
mainfrom
feat/capacity-control
Aug 9, 2026
Merged

Close M17, and make EVAL.md readable in order#12
jadoon200 merged 2 commits into
mainfrom
feat/capacity-control

Conversation

@jadoon200

Copy link
Copy Markdown
Owner

Three loose ends: a stale private plan, an untested milestone, and a document grown past the
point where it could be trusted to agree with itself.

M17 / Q3 — a foundation model changes nothing that matters

Chronos-Bolt zero-shot, Apache-2.0 per the licence audit, kept as an optional extra
torch has no place in a container that serves a precomputed snapshot.

The ceiling is the finding. Chronos-Bolt is trained on quantile levels 0.1–0.9 and
cannot express p95 or p99. It clamps to p90 and warns rather than failing, so a newsvendor
sizer asking for p95 receives p90 wearing a p95 label — worse than an error, because it looks
like an answer and nothing downstream can detect it. For a project whose thesis is that a
controller consumes a calibrated quantile, the most capable forecaster available cannot
supply the object the decision layer needs. Pinned by test.

Accuracy — marginal, exactly as registered. Wins 1 MASE row of 5; loses to plain
seasonal-naive on 3.

The decision — identical everywhere. Not one cell changes; it reproduces the classical
result on every workload including the materna-2 exception.

workload classical best chronos-bolt
materna-2 2/4 2
the other four 0/4 0

Five forecaster families have now been tested against the same decision and none moves it.
That retires the caveat left on 08-08 that "most sophisticated in this repository" wasn't
the field's best. The commitment result is a property of the demand, not of the predictor.

EVAL.md, read as a hostile reviewer

Three genuine incoherences, all from sections written days apart:

  1. The opening paragraph still said the newsvendor metric would arrive "once M4–M6 exist"
    — a week after they did.
  2. A line in the 08-07 section still said Q13 remained open, which two later sections
    refute.
  3. The M4 gate section presented a green verdict with no hint that this exact gate passed
    for five days while missing the actuation bug. That's the single most important
    cross-reference in the document and it was absent.

All three now carry forward pointers, and the document opens with a map: it is chronological,
later sections overturn earlier ones, superseded tables are banner-marked, and every question
is listed with where it ended up. Internal anchors verified to resolve.

Engineering note

Installing the extra made the suite segfault — torch and lightgbm each load an OpenMP
runtime and on macOS the pair crashes the interpreter mid-run (201 pass without, 3 pass alone,
both together die). Pinned to one thread rather than reaching for KMP_DUPLICATE_LIB_OK,
which suppresses the duplicate-runtime check and can corrupt memory silently. Foundation tests
are marked and excluded from the default run; CI never installs the extra and skips cleanly.

Result

Served findings 9 → 10. .claude/PLAN.md rewritten to match reality (gitignored, so not
in this diff).

Gate

ruff, format, mypy, pytest, validate-simulator, make evaluate-foundation — all verified by
exit code.

Three loose ends: a stale private plan, an untested milestone, and a document
that had grown past the point where it could be trusted to agree with itself.

M17 is done. Chronos-Bolt zero-shot, Apache-2.0 per the licence audit, kept as an
optional extra because torch has no place in a container that serves a
precomputed snapshot. It wins one accuracy row of five and loses to plain
seasonal-naive on three, which is "marginally at best" exactly as registered.

The ceiling is the real finding. Chronos-Bolt is trained on quantile levels
0.1-0.9 and cannot express p95 or p99 — it clamps to p90 and warns rather than
failing, so a newsvendor sizer asking for p95 receives p90 wearing a p95 label.
That is worse than an error because it looks like an answer, and nothing
downstream can detect it. For a project whose thesis is that a controller
consumes a calibrated quantile rather than a point estimate, the most capable
forecaster available cannot supply the object the decision layer needs.

On the decision itself, not one cell changes. Chronos reproduces the classical
result on every workload including the materna-2 exception. Five forecaster
families have now been tested against the same decision and none of them moves
it, which retires the caveat left on 08-08 that "most sophisticated in this
repository" was not the same as the field's best. The commitment result is a
property of the demand, not of the predictor.

Installing the extra made the suite segfault: torch and lightgbm each load an
OpenMP runtime and on macOS the pair crashes the interpreter mid-run. Pinned to
one thread rather than reaching for KMP_DUPLICATE_LIB_OK, which suppresses the
duplicate-runtime check and can corrupt memory silently. The foundation tests are
marked and excluded from the default run; CI never installs the extra and skips
them cleanly.

Read EVAL.md end to end as a hostile reviewer would. Three genuine incoherences,
all from sections written days apart. The opening paragraph still spoke of the
newsvendor metric arriving "once M4-M6 exist", a week after they did. A line in
the 08-07 section still said Q13 remained open, which two later sections refute.
And the M4 gate section presented a green verdict with no hint that this exact
gate passed for five days while missing the actuation bug — the single most
important cross-reference in the document, and it was absent. All three now carry
forward pointers, and the document opens with a map saying it is chronological,
that later sections overturn earlier ones, and where each question ended up.

The private plan is rewritten to match reality. It had sat claiming M5 was in
progress and M9-M24 unstarted while the product was deployed and serving. It is
gitignored and misled nobody but its author, who is still the person it most
needs to be accurate for.
CI installs neither chronos nor torch, so mypy could not resolve torch there while
resolving it locally — the gate passed on my machine and failed on a clean one.
Both extras are now in the scoped override, and the fix was verified in a fresh
environment built from requirements-dev alone rather than by pushing and hoping.
@jadoon200
jadoon200 merged commit 7c3b23e into main Aug 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant