Close M17, and make EVAL.md readable in order - #12
Merged
Conversation
Three loose ends: a stale private plan, an untested milestone, and a document that had grown past the point where it could be trusted to agree with itself. M17 is done. Chronos-Bolt zero-shot, Apache-2.0 per the licence audit, kept as an optional extra because torch has no place in a container that serves a precomputed snapshot. It wins one accuracy row of five and loses to plain seasonal-naive on three, which is "marginally at best" exactly as registered. The ceiling is the real finding. Chronos-Bolt is trained on quantile levels 0.1-0.9 and cannot express p95 or p99 — it clamps to p90 and warns rather than failing, so a newsvendor sizer asking for p95 receives p90 wearing a p95 label. That is worse than an error because it looks like an answer, and nothing downstream can detect it. For a project whose thesis is that a controller consumes a calibrated quantile rather than a point estimate, the most capable forecaster available cannot supply the object the decision layer needs. On the decision itself, not one cell changes. Chronos reproduces the classical result on every workload including the materna-2 exception. Five forecaster families have now been tested against the same decision and none of them moves it, which retires the caveat left on 08-08 that "most sophisticated in this repository" was not the same as the field's best. The commitment result is a property of the demand, not of the predictor. Installing the extra made the suite segfault: torch and lightgbm each load an OpenMP runtime and on macOS the pair crashes the interpreter mid-run. Pinned to one thread rather than reaching for KMP_DUPLICATE_LIB_OK, which suppresses the duplicate-runtime check and can corrupt memory silently. The foundation tests are marked and excluded from the default run; CI never installs the extra and skips them cleanly. Read EVAL.md end to end as a hostile reviewer would. Three genuine incoherences, all from sections written days apart. The opening paragraph still spoke of the newsvendor metric arriving "once M4-M6 exist", a week after they did. A line in the 08-07 section still said Q13 remained open, which two later sections refute. And the M4 gate section presented a green verdict with no hint that this exact gate passed for five days while missing the actuation bug — the single most important cross-reference in the document, and it was absent. All three now carry forward pointers, and the document opens with a map saying it is chronological, that later sections overturn earlier ones, and where each question ended up. The private plan is rewritten to match reality. It had sat claiming M5 was in progress and M9-M24 unstarted while the product was deployed and serving. It is gitignored and misled nobody but its author, who is still the person it most needs to be accurate for.
CI installs neither chronos nor torch, so mypy could not resolve torch there while resolving it locally — the gate passed on my machine and failed on a clean one. Both extras are now in the scoped override, and the fix was verified in a fresh environment built from requirements-dev alone rather than by pushing and hoping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three loose ends: a stale private plan, an untested milestone, and a document grown past the
point where it could be trusted to agree with itself.
M17 / Q3 — a foundation model changes nothing that matters
Chronos-Bolt zero-shot, Apache-2.0 per the licence audit, kept as an optional extra —
torch has no place in a container that serves a precomputed snapshot.
The ceiling is the finding. Chronos-Bolt is trained on quantile levels 0.1–0.9 and
cannot express p95 or p99. It clamps to p90 and warns rather than failing, so a newsvendor
sizer asking for p95 receives p90 wearing a p95 label — worse than an error, because it looks
like an answer and nothing downstream can detect it. For a project whose thesis is that a
controller consumes a calibrated quantile, the most capable forecaster available cannot
supply the object the decision layer needs. Pinned by test.
Accuracy — marginal, exactly as registered. Wins 1 MASE row of 5; loses to plain
seasonal-naive on 3.
The decision — identical everywhere. Not one cell changes; it reproduces the classical
result on every workload including the
materna-2exception.materna-2Five forecaster families have now been tested against the same decision and none moves it.
That retires the caveat left on 08-08 that "most sophisticated in this repository" wasn't
the field's best. The commitment result is a property of the demand, not of the predictor.
EVAL.md, read as a hostile reviewer
Three genuine incoherences, all from sections written days apart:
— a week after they did.
refute.
for five days while missing the actuation bug. That's the single most important
cross-reference in the document and it was absent.
All three now carry forward pointers, and the document opens with a map: it is chronological,
later sections overturn earlier ones, superseded tables are banner-marked, and every question
is listed with where it ended up. Internal anchors verified to resolve.
Engineering note
Installing the extra made the suite segfault — torch and lightgbm each load an OpenMP
runtime and on macOS the pair crashes the interpreter mid-run (201 pass without, 3 pass alone,
both together die). Pinned to one thread rather than reaching for
KMP_DUPLICATE_LIB_OK,which suppresses the duplicate-runtime check and can corrupt memory silently. Foundation tests
are marked and excluded from the default run; CI never installs the extra and skips cleanly.
Result
Served findings 9 → 10.
.claude/PLAN.mdrewritten to match reality (gitignored, so notin this diff).
Gate
ruff, format, mypy, pytest,
validate-simulator,make evaluate-foundation— all verified byexit code.