Accuracy Program Phase A — a gate that separates skill from luck - #3
Open
xeri wants to merge 8 commits into
Open
Accuracy Program Phase A — a gate that separates skill from luck#3xeri wants to merge 8 commits into
xeri wants to merge 8 commits into
Conversation
AGENTS.md/CLAUDE.md, the three path-scoped rules, the three hooks and the three tracking docs have been running out of the working tree untracked. They are the guardrails the repo is actually worked under, so they belong in it. accuracy-a moves to active in the same commit: Phase A starts here.
The aggregate had only ever been measured by MAE, which is why nobody noticed it loses to its own naive (6.03 vs 4.82). It now carries a predictive and is scored by the same proper rule as everything else: CRPS 4.32 against the naive's 3.18, i.e. a skill of -0.36.
…istory A paired test needs the baseline's per-fold scores; v1 stored only aggregates. The engine is unchanged — skill stays 0.2598, cover90 0.8214. Deviation from the plan, and the reason: the plan keys a fold on subjectId|targetDate. That is not unique on the committed fixture, which has 24 days where a desk both sits an exam and hands in coursework — 56 folds collapse to 38 keys and the paired comparison differences half the book against the wrong partner. OneStep gains targetId and the key is the triple. Caught by the plan's own 'emits unique keys' assertion, which is why it was written. eval.book.test.ts is red until the next commit, which is the gate rewrite that reads the new shape. npx tsc --noEmit is red on the same two lines.
Replaces skill > baseline - 0.03, which was one-sided, had no notion of sampling error, and could not have had one (v1 stored no per-fold data). Verified failable, which is the only check that matters here: with ENSEMBLE_DISPERSION raised 1.15 -> 3.0 the gate reports REGRESSED and the best-historical floor trips too (skill 0.0769 against a floor of 0.2098); reverted, all 63 gate tests pass and the verdict is INDISTINGUISHABLE. Adds a best-historical floor and an MDE readout, and records in README §26 that the aggregate loses to its own naive (CRPS 4.32 vs 3.18) and that this book cannot resolve a CRPS gain below ~2.12 points against a mean of 6.78. The change protocol gains step 8: gen:baseline is its own command now, so step 4's claim that gen:table regenerates the baseline is deleted.
Gives the harness a sample size the one committed fixture cannot. Decides estimator structure and finds bugs; never tunes a constant — the rule is in the module header and it is the reason the module exists at all. First finding, over 840 folds from 30 books: on data drawn from the model's OWN assumptions — local-level random walk, per-type observation noise, no paper-difficulty term — realized 90% coverage is 0.864, not 0.90. The under-coverage is therefore structural rather than a symptom of the real book's misspecification. Adding an unmodelled difficulty shock (tau = 8) takes it to 0.839, so the harness can see misspecification when it is there, which is what Phase C's difficulty work has to be adjudicated on. Uses pDate rather than the plan's new Date(iso): the latter parses as UTC and would disagree with addDays, which is local, so the generated gaps would not match the generated dates.
accuracy-a closes. agent-notes carries the new clean-tree numbers (1359/2, 108 files) and the two facts a fresh session will otherwise get wrong: the gate is a three-way verdict, and baseline.json has its own regeneration command. Both files now state the MDE — +/-2.12 CRPS points against a mean of 6.78 — because it is the number that decides whether a proposal is worth making at all. New candidate: the ensemble's moment match, settled on synthetic books. Phase A measured 0.864 realized coverage on data drawn from the model's own assumptions, so the under-coverage is arithmetic rather than misspecification, and the fixture cannot separate the audit's Sec 10.2 fix from the swept ENSEMBLE_DISPERSION that is absorbing it. Synthetic books can.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
docs/superpowers/plans/2026-07-31-accuracy-program-phase-a.mdend to end. Closes board rowaccuracy-a.Phase A changes the measuring instrument, never a forecast. Per-subject skill is 0.2598 before and after, and
git diff main -- src/lib/quant/ ':!src/lib/quant/eval'is empty.What lands
eval/rng.ts— the one seeded PRNG the engine is allowed (mulberry32), so the bootstrap and the generator are pure functions of an integer.eval/compare.ts+eval/params.ts— paired cluster bootstrap over subjects, three-way verdict (IMPROVED/INDISTINGUISHABLE/REGRESSED), and a minimum-detectable-effect bound. Folds within a subject share a tape, so resampling folds individually would report an interval several times too narrow.eval/skill.ts— the aggregate target now carries a predictive and is scored by CRPS and coverage, not MAE alone.eval/snapshot.ts+gen-baseline.ts— baseline v2 with a per-fold vector, an append-onlyhistory.json, andnpm run gen:baseline -- "<note>"(the note is required).eval.book.test.ts— the gate itself, plus a best-historical skill floor and an MDE readout.eval/synth.ts— generative books with known truth, for the sample size one fixture cannot give.What the new instrument immediately reported
Both of these were invisible to the old gate and neither is a change made here:
q, per-type noise, no unmodelled difficulty), realized 90% coverage is 0.864. There is nothing left for misspecification to explain.Verification
npm run gate— 70 passed, 3.5s (budget was 15s).ENSEMBLE_DISPERSIONraised 1.15 → 3.0 it reportsREGRESSEDand the historical floor trips too (skill 0.0769 against a floor of 0.2098). Reverted, green.npm test— 1366 passed, 2 failed:SpiderAllocator.test.tsx(stale brand hex) andderive.book.test.ts(KaTeX 5s timeout). Both pre-existing onmain, both tracked as loose ends on the board.npx tsc --noEmit— clean.src/lib/__fixtures__/book.jsonunmodified;npm run gen:tablenever run.One deviation from the plan
The plan keys a fold on
subjectId|targetDate. That is not unique on the committed fixture — 24 days have a desk both sitting an exam and handing in coursework, so 56 folds collapse to 38 keys and the paired comparison differences half the book against the wrong partner.OneStepgainstargetIdand the key is the triple. The plan's own "emits unique keys" assertion is what caught it.Also uses
pDaterather than the plan'snew Date(iso)insynth.ts: the latter parses as UTC and would disagree withaddDays, which is local.Sequencing
Phase A had to precede the audit's §8 (
fit.ts+earned.ts), because each channel conversion is gated against the restructured baseline.earned.tsis still on the old score-share estimator, so that ordering held.