Skip to content

Close the pre-registered question ledger - #11

Merged
jadoon200 merged 1 commit into
mainfrom
feat/capacity-control
Aug 9, 2026
Merged

Close the pre-registered question ledger#11
jadoon200 merged 1 commit into
mainfrom
feat/capacity-control

Conversation

@jadoon200

Copy link
Copy Markdown
Owner

Three of the ten registered questions were unaccounted for. For a project whose central claim
is "pre-registered questions, answered whichever way they fell", that's the gap most worth
closing.

Q9 — never measured, and the ROADMAP implied otherwise

generate_multi_resource existed and was unit-tested, but no experiment used it, while
docs/ROADMAP.md claimed the synthetic regime "carries the joint-provisioning test". It
didn't. Corrected rather than quietly deleted.

The registered direction is refuted. Independent per-resource sizing under-provisions,
and the sign follows from max(a,b) >= a without needing an experiment — the joint quantile is
never below either marginal, so combining independently sized resources can only buy less.

What's actually measured is the coverage it costs (target q = 0.90):

offset correlation joint cov. indep. cov.
0.000 +0.778 0.9688 0.8442
0.250 +0.010 0.9688 0.8229
0.500 −0.770 0.9688 0.8016

The shortfall widens monotonically as the resources decorrelate — the more the peaks avoid
each other, the more often the resource you aren't watching is the binding one. You cannot
reach a joint SLO from one dashboard per resource.

Three nulls first, all kept in the record

A null from an instrument that cannot detect the effect is not evidence of absence, so the
sequence is documented: peaks narrower than the tail a quantile discards; peaks landing in
only one half of the train/test split; and a median-based replica scale that let the
wider-swinging resource bind at every step. The tell each time was a null that was too
clean
— identical to four decimals across conditions that should have differed. The generator
gained peak-width and diurnal-phase parameters, both defaulting to the original fixture.

Q2 — the comparator was a strawman

It measured conformal against an uncalibrated forecast. It now carries the fixed safety
margin it registered, both tuned and at the conventional +15% VPA ships:

Method p95 coverage Mean p95 Recovery
fixed_margin_conventional_+15% 0.917 113.563 46
aci_gamma_0.05 0.924 101.976 28

ACI holds better coverage on 11% less capacity and recovers in 28 steps where a fixed
margin never recovers — it can't, it doesn't respond to anything. The tuned margin is sharper
still: raw p95 already over-covers on stationary validation, so the honest tuned margin is
+0%, offering no protection whatever when the level shifts.

Q6 — measured, never labelled

The GPU structural claim is the answer to Q6. It was carried out under its milestone number
and never tied back — a bookkeeping failure in a project whose discipline is that ledger.
Labelled and shipped.

Result

Served findings go from 6 to 9. Makefile targets added for all four new experiments.

Gate

ruff, format, mypy, pytest, validate-simulator — all verified by exit code. 201 tests, M4
gate GREEN.

Three of the ten registered questions were unaccounted for. For a project whose
central claim is "pre-registered questions, answered whichever way they fell",
that is the gap most worth closing.

Q9 had never been measured at all, while docs/ROADMAP.md claimed the synthetic
multi-resource regime "carries the joint-provisioning test". It did not — the
generator existed and was unit-tested, but no experiment used it. The sentence is
corrected rather than quietly deleted.

The registered direction turns out to be wrong. Independent per-resource sizing
under-provisions rather than over-provisions, and the sign follows from
max(a,b) >= a without needing an experiment: the quantile of the joint requirement
is never below either marginal, so combining independently sized resources can
only buy less. What is measured is the coverage that costs. Asked for 90%, the
independent method delivers 84.4% when CPU and memory move together and 80.2% when
they are anti-phase, against a joint plan's steady 96.9%, widening monotonically as
correlation falls from +0.78 to -0.77.

Three designs returned exact nulls first and all three are in the record, because a
null from an instrument that cannot detect the effect is not evidence of absence.
Peaks narrower than the tail a quantile discards; peaks landing in only one half of
the train/test split; and a median-based replica scale that let the wider-swinging
resource bind at every step, so decorrelation had nothing to act on. The tell each
time was a null that was too clean — identical to four decimals across conditions
that should have differed. The generator gained peak width and diurnal phase
parameters, both defaulting to the original fixture so existing tests are untouched.

Q2 was measured only against an uncalibrated forecast, which is a strawman. It now
carries the comparator it registered: a fixed safety margin, both tuned on
validation and at the conventional +15% that VPA ships. ACI holds better coverage
on 11% less capacity and recovers in 28 steps where a fixed margin cannot recover
at all. The tuned margin is the sharper result — raw p95 already over-covers on
stationary validation, so an honestly tuned margin is +0% and offers no protection
whatever when the level shifts.

Q6 was measured under its milestone number and never tied back to the question it
answers. Labelled, and shipped as a finding.

All three now appear in the served findings, so the site states nine.
@jadoon200
jadoon200 merged commit 34f9085 into main Aug 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant