fix(eval): commit each published score to the corpus a reader can download - #9811
Merged
Conversation
…nload
Verified live on api.loopover.ai: /v1/public/stats publishes
ai_consensus_defect at decided 460 / confirmed 287, /v1/public/eval-corpus
serves 460 real cases whose checksum re-derives from the downloaded bytes, and
/v1/public/eval-scores serves `{"records":[]}`.
The commitment came only from a persisted calibration.*_backtest_run audit
event. #9639 fixes that writer, but it runs inside a review pass, and hosted
review execution is retired (src/index.ts acks-and-drops review-execution jobs
off the queue), so on loopover.ai the event never exists. The walkthrough tells
an anonymous reader to fetch .records and re-derive recordDigest; they got an
empty array, explained by a doc line attributing it to an empty corpus that
demonstrably is not empty.
Each record now falls back to its OWN rule's published corpus checksum -- the
exact bytes /v1/public/eval-corpus serves, over the same window. That is not the
placeholder commitment #9215 forbids: it is a hash over an artifact the reader
downloads and re-hashes, which is what the `reproducible` tier asserts. A
persisted run still wins where one exists, so self-host is unchanged.
Resolving per rule also fixes a latent bug: one run's checksum was stamped onto
every record, so with more than one published rule every record but one would
have committed to a different rule's cases.
TRUNCATION HAD TO BE MADE DETECTABLE FIRST. PUBLIC_EVAL_CORPUS_MAX_CASES was
5_000 while the corpus is built from a rule-history read that
listAuditEventsByType hard-clamps to 2_000 -- and queryRuleHistory passed no
limit at all, so it took the default of 500. The cap could never bind, and
`truncated` was structurally always false, while /v1/public/stats counts
`decided` with an unbounded SQL COUNT(*). The two surfaces agreed only while the
window stayed under 500 cases; at 460 they were 40 from silently diverging, with
the corpus serving a prefix and reporting completeness. queryRuleHistory now
takes an explicit bound and reports `saturated`, the corpus ORs that into
`truncated`, and a truncated corpus is never published as a commitment -- its
checksum would cover a prefix of the window the score covers.
Closes #9805
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 3e3e603 | Commit Preview URL Branch Preview URL |
Jul 29 2026, 10:56 AM |
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-29 11:19:15 UTC
Review summary Nits — 2 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Bundle ReportChanges will increase total bundle size by 4.5kB (0.06%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9811 +/- ##
=======================================
Coverage 90.46% 90.47%
=======================================
Files 918 918
Lines 113940 113970 +30
Branches 27053 27060 +7
=======================================
+ Hits 103079 103109 +30
Misses 9533 9533
Partials 1328 1328
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9805
Verified live before and after
The corpus is real and complete, and I confirmed
checksumPublicEvalCorpusover thecasesarray exactly as downloaded reproduces470e234e…. The 460 published cases are also the same 460 thedecidedcount is computed from. Yet nothing committed to it.The commitment came only from a persisted
calibration.*_backtest_runaudit event. #9804 fixes that writer — but it runs inside a review pass, and hosted review execution is retired (src/index.tsacks-and-dropsisReviewExecutionJobmessages), so on loopover.ai that event never exists. Meanwhileverify-this-review.mdx§4 tells an anonymous reader to fetch.recordsand re-deriverecordDigest, and the doc explained the empty array as "the latest backtest run's corpus is empty" — which was no longer true.What changed
Each record falls back to its own rule's published corpus checksum: the exact bytes
/v1/public/eval-corpusserves, over the same window. That is not the placeholder commitment #9215 forbids — it is a hash over an artifact the reader downloads and re-hashes, which is precisely what thereproducibletrust tier asserts. A persisted run still wins where one exists, so self-host behaviour is untouched.Resolving per rule also fixes a latent bug:
latestBacktestRun.corpusChecksumwas a single value stamped onto every record, so with more than one published rule, every record but one would have committed to a different rule's cases. Only one rule clears the floor today, which is why it hadn't surfaced.The part I did not expect to have to fix
The issue requires "a truncated corpus must not be published as a commitment." Truncation was undetectable:
PUBLIC_EVAL_CORPUS_MAX_CASESwas5_000queryRuleHistory, which passed no limit, solistAuditEventsByTypeused its default of 500 (hard-clamped at 2000 regardless)So the cap could never bind and
truncatedwas structurally alwaysfalse— while/v1/public/statscountsdecidedwith an unbounded SQLCOUNT(*). The two surfaces agreed only while the window stayed under 500 cases. At 460, they were 40 cases from silently diverging:statsreporting the true count while the corpus served a 500-case prefix and reported completeness.That directly undermines this PR's commitment, so it is fixed here rather than deferred:
queryRuleHistorytakes an explicitlimit(defaulted, so every existing caller is byte-identical) and returnssaturated— true when either read came back at its boundPUBLIC_EVAL_CORPUS_MAX_CASESis pinned to the real ceiling instead of an unreachable numbertruncatedORs the cap with read saturationThe interface change ripples to
SignalStoreand the miner's own store. The miner reads its whole local event ledger in memory with no bound, so it returnssaturated: false— a true statement there, not a stub.Deliverables
/v1/public/eval-corpuspayload for the same rule, end to end through both routes withrecordDigestre-verifiedverify-this-review.mdx§4 rewritten: it now gives the reader the two-command loop to tie a record to the corpus they downloaded, and states the three real reasons a rule is omittedValidation
tsc --noEmit·db:schema-drift:check·dead-source-files:check·import-specifiers:check·validate:no-hand-written-js·docs:drift-check— clean. 176 tests pass across the affected and ripple-affected suites.Coverage on every changed file —
eval-score-records.ts,public-eval-corpus.ts,signal-tracking-wire.ts— 100% lines, 100% branches.Note on ordering
Independent of #9804; they touch different sides of the same surface and merge in either order. Together they mean a deployment publishes records whether or not it executes reviews.
Documentation-only change to
apps/loopover-ui/content/docs/; no route, component, or style touched, so there is no visual diff to screenshot.