Close the pre-registered question ledger - #11
Merged
Conversation
Three of the ten registered questions were unaccounted for. For a project whose central claim is "pre-registered questions, answered whichever way they fell", that is the gap most worth closing. Q9 had never been measured at all, while docs/ROADMAP.md claimed the synthetic multi-resource regime "carries the joint-provisioning test". It did not — the generator existed and was unit-tested, but no experiment used it. The sentence is corrected rather than quietly deleted. The registered direction turns out to be wrong. Independent per-resource sizing under-provisions rather than over-provisions, and the sign follows from max(a,b) >= a without needing an experiment: the quantile of the joint requirement is never below either marginal, so combining independently sized resources can only buy less. What is measured is the coverage that costs. Asked for 90%, the independent method delivers 84.4% when CPU and memory move together and 80.2% when they are anti-phase, against a joint plan's steady 96.9%, widening monotonically as correlation falls from +0.78 to -0.77. Three designs returned exact nulls first and all three are in the record, because a null from an instrument that cannot detect the effect is not evidence of absence. Peaks narrower than the tail a quantile discards; peaks landing in only one half of the train/test split; and a median-based replica scale that let the wider-swinging resource bind at every step, so decorrelation had nothing to act on. The tell each time was a null that was too clean — identical to four decimals across conditions that should have differed. The generator gained peak width and diurnal phase parameters, both defaulting to the original fixture so existing tests are untouched. Q2 was measured only against an uncalibrated forecast, which is a strawman. It now carries the comparator it registered: a fixed safety margin, both tuned on validation and at the conventional +15% that VPA ships. ACI holds better coverage on 11% less capacity and recovers in 28 steps where a fixed margin cannot recover at all. The tuned margin is the sharper result — raw p95 already over-covers on stationary validation, so an honestly tuned margin is +0% and offers no protection whatever when the level shifts. Q6 was measured under its milestone number and never tied back to the question it answers. Labelled, and shipped as a finding. All three now appear in the served findings, so the site states nine.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three of the ten registered questions were unaccounted for. For a project whose central claim
is "pre-registered questions, answered whichever way they fell", that's the gap most worth
closing.
Q9 — never measured, and the ROADMAP implied otherwise
generate_multi_resourceexisted and was unit-tested, but no experiment used it, whiledocs/ROADMAP.mdclaimed the synthetic regime "carries the joint-provisioning test". Itdidn't. Corrected rather than quietly deleted.
The registered direction is refuted. Independent per-resource sizing under-provisions,
and the sign follows from
max(a,b) >= awithout needing an experiment — the joint quantile isnever below either marginal, so combining independently sized resources can only buy less.
What's actually measured is the coverage it costs (target q = 0.90):
The shortfall widens monotonically as the resources decorrelate — the more the peaks avoid
each other, the more often the resource you aren't watching is the binding one. You cannot
reach a joint SLO from one dashboard per resource.
Three nulls first, all kept in the record
A null from an instrument that cannot detect the effect is not evidence of absence, so the
sequence is documented: peaks narrower than the tail a quantile discards; peaks landing in
only one half of the train/test split; and a median-based replica scale that let the
wider-swinging resource bind at every step. The tell each time was a null that was too
clean — identical to four decimals across conditions that should have differed. The generator
gained peak-width and diurnal-phase parameters, both defaulting to the original fixture.
Q2 — the comparator was a strawman
It measured conformal against an uncalibrated forecast. It now carries the fixed safety
margin it registered, both tuned and at the conventional +15% VPA ships:
fixed_margin_conventional_+15%aci_gamma_0.05ACI holds better coverage on 11% less capacity and recovers in 28 steps where a fixed
margin never recovers — it can't, it doesn't respond to anything. The tuned margin is sharper
still: raw p95 already over-covers on stationary validation, so the honest tuned margin is
+0%, offering no protection whatever when the level shifts.
Q6 — measured, never labelled
The GPU structural claim is the answer to Q6. It was carried out under its milestone number
and never tied back — a bookkeeping failure in a project whose discipline is that ledger.
Labelled and shipped.
Result
Served findings go from 6 to 9. Makefile targets added for all four new experiments.
Gate
ruff, format, mypy, pytest,
validate-simulator— all verified by exit code. 201 tests, M4gate GREEN.