feat(fairness): one-command public verifier + published methodology page - #9941
Merged
Conversation
Closes #9723. LoopOver publishes fairness numbers and says a stranger can check them, but checking them meant reimplementing canonical JSON, reimplementing ECDSA over it, and hand-diffing four endpoints -- so in practice nobody did, and a broken commitment could sit published indefinitely. npx -p @loopover/mcp loopover-verify Fetches the public surfaces, recomputes every commitment, prints a per-claim PASS/FAIL table and exits non-zero on any failure. Four claims: every recordDigest recomputes from its own record; every corpusChecksum rehashes to a downloadable corpus, with a scored record committing to the EMPTY corpus treated as a failure; the signed ledger checkpoint verifies offline against a published key, including that the displayed signingInput really is the preimage of the displayed payload; and the headline totals agree with the ledger-derived parity rollups. No credentials of any kind: every path is under /v1/public/ and no Authorization header is ever set, asserted by a test. To make that possible without a repo checkout, the digest primitives (canonicalJson/sha256Hex/contentDigest) and the VERIFYING half of ledger anchoring move into @loopover/contract, re-exported from their old homes so every call site is unchanged. Signing stays in the Worker, which is the only thing that may hold the private key. The anchor verifier's own comment used to say it was 'the function a third party reimplements' -- that reimplementation was the gap. A claim that could not be checked reports SKIP and is never counted as a pass. The api-schema generator now scans the bin directory rather than one hardcoded filename, so a future bin cannot silently opt out of response validation. Output is byte-identical today.
Closes #9725. /fairness publishes precision, accuracy and parity figures whose method lived in code and issue history, so a reader could see the number but not the rule behind it. This is that page: the three disjoint data sources and which figures come from each, the outcome-confirmation rules, every window, precision and coverage definitions, sample floors, and a limitations section. Linked from /fairness (both the rule table and the parity block) and from the verify-this-review walkthrough, and kept next to the scoring code it describes so a definition change and a doc change land in the same review. Written against the code rather than from intent, which surfaced things worth stating plainly: the windows are flat with no time-decay anywhere; only the review-parity block is computed from the anchored ledger; coverage is computed but not displayed here; and the only interval on the page is the headline accuracy CI, in the single-instance branch -- per-rule precision, the per-repository column, both weekly trends and every parity figure are bare point estimates. Also fixes a mislabel the write-up exposed: the headline read "lifetime" while its accuracy pairing has always been bounded by audit-log retention, deliberately so (a lifetime denominator drifts the ratio toward 100%). totals now publish accuracyWindowDays, derived from the retention policy rather than restated as a literal, and the page renders the real window.
Contributor
|
Important 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏳ LoopOver is waiting…LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | ac3e597 | Commit Preview URL Branch Preview URL |
Jul 30 2026, 06:50 PM |
Bundle ReportChanges will increase total bundle size by 36.64kB (0.47%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9941 +/- ##
==========================================
- Coverage 91.87% 91.87% -0.01%
==========================================
Files 923 927 +4
Lines 113436 113572 +136
Branches 27331 27377 +46
==========================================
+ Hits 104219 104342 +123
- Misses 7928 7931 +3
- Partials 1289 1299 +10
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9723
Closes #9725
Epic #9722. Two halves of the same goal: make the fairness numbers checkable by a stranger, and make the method behind them readable without opening the source.
The verifier (#9723)
Fetches the public surfaces, recomputes every commitment, prints a per-claim PASS/FAIL table, exits non-zero on any failure. Four claims:
record-digestsrecordDigest, from the record minus the digest, under the same canonical serializationcorpus-commitmentscorpusChecksum, rehashed from the served cases — not trusted from the corpus's own checksum field. A record committing to the EMPTY corpus while reportingdecided > 0is a failureanchor-checkpointsigningInputreally is the preimage of the displayed payloadstats-parityZero credentials: every path is under
/v1/public/, noAuthorizationheader is ever set, and no token is read from the environment. There is a test asserting that on every request.A claim that could not be checked reports SKIP and is never counted as a pass. Skips do not fail the run — a disabled surface is not a broken one — but the two are kept structurally distinct all the way to the exit code. (An earlier draft reported PASS with a detail line reading "unverified"; the regression test for that is in the suite.)
Why the contract package moved
For an outsider to recompute a digest they need the serialization rule, and it lived inside the Worker. So
canonicalJson/sha256Hex/contentDigest, and the verifying half of ledger anchoring, move into@loopover/contractand are re-exported from their old homes — every existing call site is unchanged, and there is one definition rather than two that can drift.Signing stays in the Worker; it needs the private key. The anchor verifier's own comment used to describe itself as "the function a third party's own verifier reimplements" — that reimplementation was the gap.
Behaviour-preserving: the existing digest and anchor suites pin concrete hex values and signature round-trips, and pass unchanged.
The methodology page (#9725)
/docs/fairness-methodology, linked from/fairness(rule table + parity block) and from the verify-this-review walkthrough.Written against the code rather than from intent, which is why it says some things a marketing page would not:
Mislabel fixed in passing
The headline read "lifetime" while its accuracy pairing has always been bounded by audit-log retention — deliberately so, since a lifetime denominator drifts the ratio toward 100%.
totalsnow publishaccuracyWindowDays, derived from the retention policy rather than restated as a literal, and the page renders the real window.Also
gen-contract-api-schemasscans the bin directory rather than one hardcoded filename, so a future bin cannot silently opt out of response validation. Output is byte-identical today — a latent fix, not a live one.Validation
test/unit+test/integration: 25,677 passed, 0 failuresui:lint/ui:typecheck/ui:buildcleanui:openapi:check,ui-derived-types:check,contract:api-schemas:check,docs:drift-check,import-specifiers:check,dead-exports:check,typecheckall pass#9724(nightly walkthrough CI job) is deliberately not included — its acceptance requires a deliberately-broken staging config turning the job red within one cycle, which is a production failure drill rather than a code change.