feat(api): publish the eval corpus so a stranger can actually verify a review - #9764
Merged
Conversation
…a review The verifiability walkthrough tells a skeptic to "export the same corpus the numbers come from", but the only exporter is scripts/backtest-corpus-export.ts, which shells out to `wrangler d1 execute --remote` and needs that deployment's own Cloudflare credentials. Step 1 was never runnable by a third party, which made the page's "nothing needs an API key" claim false. #9636 fixed the claim by marking the step operator-only; this fixes the gap. GET /v1/public/eval-corpus?ruleId=<id> serves one rule's labeled corpus over the same window the published per-rule precision covers, unauthenticated. REDACTION IS THE DESIGN, and it costs nothing that replay needs: - `targetKey` (literally `owner/repo#number`) is DROPPED, not hashed. scoreBacktest never reads it, and a hash would still be a stable per-PR identifier -- correlatable, for no replay benefit. - metadata is narrowed to `confidence`, kept NESTED where buildConfidenceThresholdClassifier actually reads it (backtest-threshold.ts:21). Flattening it to a top-level field would make every replay silently hit the `?? 1` fallback and label every case "confirmed" -- a wrong answer that looks like a working one. - timestamps are truncated to the UTC day. Nothing in the replay path reads them; full-precision pairs were the one remaining way to line a case up against a private repo's PR timeline. The checksum commits to THIS artifact, not to the internal manifest. A reader can only re-derive what they can download, and a checksum over a preimage nobody can obtain verifies nothing -- exactly the failure #9636 fixed on the eval-score records. It deliberately uses canonicalJson + Web Crypto, the same pair every other digest here uses, so a consumer learns one serialization rule. Counterfactual-replay overrides are excluded, matching the per-rule precision this corpus explains -- otherwise the corpus and the number would disagree. Docs: step 1 is now a credential-free curl, and step 2's verification snippet imports NOTHING from LoopOver -- canonical JSON is eight lines, and running our code to check our numbers is not much of a check. The snippet was executed against real output before being written down. A callout records that the operator CLI's older canonicalization produces a different, deliberately non-comparable checksum, since each commits to the artifact it ships with. what-you-can-verify.mdx's "private repositories are the exception" callout is corrected rather than left flattering: a private repo's cases DO appear here, because the published precision counts them and a corpus omitting them would not explain those numbers. They appear stripped of identity. What is still not published is any identifiable per-tenant history. Closes #9636
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-29 09:37:50 UTC
Review summary Nits — 7 non-blocking
Concerns raised — review before merging
📋 Copy for AI agents — paste into your coding agentDecision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionNot yet addressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | aa5fbb1 | Commit Preview URL Branch Preview URL |
Jul 29 2026, 09:19 AM |
Bundle ReportChanges will increase total bundle size by 9.27kB (0.12%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9764 +/- ##
==========================================
- Coverage 90.33% 89.49% -0.85%
==========================================
Files 917 917
Lines 113904 113643 -261
Branches 26966 26959 -7
==========================================
- Hits 102894 101703 -1191
- Misses 9681 10850 +1169
+ Partials 1329 1090 -239
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Owner
Author
|
Not previously fixed — you were right to push, and it turned out to be worse than the reviewer framed it. Two defects, both now fixed in the latest push. 1. The fallback did mask an outage — and it was the exact bug this endpoint exists to close
That is a checksummed empty corpus. It verifies perfectly and commits to nothing — precisely the defect #9636 was opened for, reintroduced by the endpoint built to close it. A reader downloading during an outage would verify the checksum, replay zero cases, and conclude the rule has no history. The siblings' degrade-to-empty posture is right for them: each is one section of a larger payload. Here the whole response is the artifact and it carries a commitment, so the read now propagates and the route returns 503 2. Worse, and not in the review:
|
| File | Changed lines | Uncovered | Partial branches |
|---|---|---|---|
public-eval-corpus.ts |
157 | none | none |
signal-tracking-wire.ts |
37 | none | none |
routes.ts |
26 | none | none |
route-auth.ts |
4 | none | none |
54 tests green, including two new regressions: the loader now rejects.toThrow() on a broken store, and the route returns 503; plus a real-D1 test seeding past the event ceiling to prove truncated flips. Rebased onto current main; typecheck and ui:openapi:check clean. The 503 is registered in OpenAPI with the reason.
# Conflicts: # apps/loopover-ui/public/openapi.json
JSONbored
added a commit
that referenced
this pull request
Jul 29, 2026
…er claim a review was skipped when it ran (#9800) Two defects found live on the ORB, both surfaced by the maintainer's own PRs: 1. The machine-paced cadence leg of shouldSkipAiForReputation downgraded the repo OWNER to deterministic-only on every PR: 50 submissions in 24h at a 7.2-minute median gap trips isMachinePacedCadence (>=5 in 24h, median gap under 10 minutes), regardless of a 145-merge success record. The cadence check is a spend control against UNPROVEN accounts; "trusted" is this same module's own definition of proven (5+ recent merged successes at a low fail rate). A trusted submitter now short-circuits the cadence read entirely. The exemption cannot mask genuine abuse: any prompt-injection row in the window hard-locks the signal to "low" before success counting, pinned by an invariant test. 2. maybeAddReputationSkipHold pushed the "Required AI review was skipped by a submitter-reputation downgrade" finding whenever the skip WOULD fire -- without checking that aiReviewAllAuthors or a forced re-run bypassed it, in which case the review genuinely ran. The published panel then asserted the review was skipped directly above that review's own summary, and held the gate for a review that succeeded (observed on #9764). The hold now describes what happened, not what the heuristic wanted. Operationally: LOOPOVER_REVIEW_REPUTATION is commented out on the ORB until this ships in an image; re-enabling is a one-line uncomment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The verifiability walkthrough tells a skeptic to "export the same corpus the numbers come from" — but the only exporter is
scripts/backtest-corpus-export.ts, which shells out towrangler d1 execute --remoteand needs that deployment's own Cloudflare credentials. Step 1 was never runnable by a third party. #9636 fixed the claim by marking the step operator-only; this fixes the gap.GET /v1/public/eval-corpus?ruleId=<id>serves one rule's labeled corpus over the same window the published per-rule precision covers, unauthenticated.Closes #9636
Redaction is the design — and it costs nothing replay needs
targetKey(owner/repo#number)scoreBacktestnever reads it, and a hash is still a stable per-PR identifier — correlatable, for zero replay benefitmetadataconfidence, kept nestedbuildConfidenceThresholdClassifierreadsmetadata.confidence(backtest-threshold.ts:21). Flattening it would make every replay hit the?? 1fallback and label every case"confirmed"— a wrong answer that looks like a working onefiredAt/decidedAtThe checksum commits to this artifact, not to the internal manifest. A reader can only re-derive what they can download, and a checksum over a preimage nobody can obtain verifies nothing — exactly the failure #9636 fixed on the eval-score records. It uses
canonicalJson+ Web Crypto, the same pair every other digest here uses, so a consumer learns one serialization rule rather than two.Counterfactual-replay overrides are excluded, matching the per-rule precision this corpus explains — otherwise the corpus and the number it backs would disagree.
Docs
Step 1 is now a credential-free
curl. Step 2's verification snippet imports nothing from LoopOver — canonical JSON is eight lines, and running our code to check our numbers is not much of a check. I executed that snippet against real output before writing it down; it printschecksum OK.A callout records that the operator CLI's older canonicalization produces a different, deliberately non-comparable checksum, since each commits to the artifact it ships with. Without that note, an operator verifying a downloaded document with
buildBacktestCorpusManifestwould get a falseCHECKSUM MISMATCH.what-you-can-verify.mdx's "private repositories are the exception" callout is corrected rather than left flattering: a private repo's cases do appear here, because the published precision counts them and a corpus omitting them would not explain those numbers. They appear stripped of identity. What is still not published is any identifiable per-tenant history.Validation
public-eval-corpus.tscoverageroutes.ts+route-auth.tstypecheck·ui:openapi:check·docs:drift-check·ui:lint(0 errors) ·ui:build·coverage-boltons·validate:no-hand-written-js·release-linked-versions:checkgit diff --checkRegistered in the table-driven
exemptPublicRoutesregression test whose stated purpose is "a future public route that forgets itsrequiresApiTokenexemption fails HERE, not in prod", andopenapi-security-paritypasses.Two invariant tests carry the weight: one asserts the serialized response never matches
/acme|private-widgets|4242|targetKey/iagainst a deliberately identifying seed, the other pinsmetadata.confidencestaying nested.Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.This adds a new unauthenticated route, so the negative paths are the point: 404 when
LOOPOVER_PUBLIC_STATSis off (same flag as its siblings), 400 withoutruleIdrather than silently picking a rule, an empty-but-still-checksummed corpus for an unknown rule, and a fail-safe empty corpus when the store read throws rather than a 500 on a public endpoint. The exemption is declared inroute-auth.ts— the single predicate both the auth middleware and the OpenAPI builder read — not in a second, driftable place.UI Evidence
Captured at 1280×800,
deviceScaleFactor: 2, dark build.Notes
On current production data this endpoint returns a real corpus for
ai_consensus_defectand an empty one forslop_gate_score— correct on both counts, and consistent with what the fairness table now shows after #9720.