docs: raw-LLM-vs-orc demos (worked examples + large private corpus) - #16
Open
Thormatt wants to merge 6 commits into
Open
docs: raw-LLM-vs-orc demos (worked examples + large private corpus)#16Thormatt wants to merge 6 commits into
Thormatt wants to merge 6 commits into
Conversation
A single legible claim run the same model two ways — a bare LLM call vs verify_claim — to make the aggregate F1 number concrete. demos/orc_vs_raw.py runs it live (gated); docs/demos/defensible-verification.md captures the real output with honest framing: on a frontier model the verdicts tie, so orc's value is the defensible artifact (validated citation, calibrated confidence, replayable trace, audit bundle) and the runtime invariant that drops fabricated citations (0/300 leaked, reproducible for free), not a smarter answer. Includes the published competitive table and reproduction commands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scanned DROP + FinanceBench for items where the production-shaped raw call (binary YES/NO faithfulness prompt) gets the verdict wrong while orc gets it right. financebench_id_07081 (CVS ROA): the quick raw call answers "YES" — rubber-stamping a wrong 0.04 ROA — because it commits before computing; orc's arithmetic mode computes 0.03 first, then rejects. Reframes the writeup into two honest regimes: tie on easy single-passage items (artifact value), raw breaks on hard numeric items (correctness value). Found by scanning, not invented; orc has its own misses too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The single-passage examples are the best case for a raw call (whole
source in the prompt). This adds the opposite: an 11-document fictional
internal knowledge base the model has never seen, with a fact buried in
one postmortem. A well-aligned model correctly REFUSES ("I will not
fabricate a source") — so a raw call simply cannot verify against your
private corpus; a less careful one confabulates. orc retrieves the one
relevant chunk of 18, returns CONTRADICTED / SUPPORTED at 0.99, and
cites the exact source file. demos/orc_large_corpus.py + demos/corpus/,
captured in the writeup; thesis and reproduce section updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thormatt
changed the title
A standalone page (deploys at /invest) in the existing orc brand — warm-dark, gold accent, Source Serif 4 / Inter / JetBrains Mono. Sells the runtime to investors and regulated-industry prospects, centered on the real raw-vs-orc receipts from the demo: the CVS ROA correctness gap (raw "YES" vs orc 0.03) and the private-corpus case (raw refuses vs orc cites the exact file). Sections: hero, the problem + EU AI Act clock, the proof receipts, four primitives, reproducible numbers, category positioning, pilot-first business model, and an honest "where we are" (strong engineering, zero distribution). Claims kept accurate and reproducible; responsive; noscript + safety-net fallbacks so content never depends on JS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "HHEM tokenizer-load issue" was a transformers-5.x incompatibility with HHEM-2.1-Open's vendored modeling code; pin transformers<5 in the benchmarks extra to fix it. New head_to_head.py reuses an existing Orc results.json and scores HHEM-2.1-Open on the exact same 503 items (no Orc re-spend). Result: Orc F1 0.864 vs HHEM 0.643 on identical items, threshold 0.5, standard (passage, answer) input. Verified honest: - not a context-window artifact (HHEM 0.623 on items inside its 512-tok window, 0.693 on truncated ones); - input format checked — answer-only (0.30 on FinanceBench) beats question+answer (0.13), so HHEM gets its more favorable input; - the gap is concentrated in numeric/tabular reasoning (FinanceBench 0.30, DROP 0.45) where a consistency encoder can't compute; HHEM stays competitive on RAG passages (RAGTruth 0.80). Caveat kept explicit: this stratified subsample equal-weights the hard categories, so it's tougher for HHEM than its ~0.75 full-HaluBench headline — and Orc's 0.864 is on the same harder set. Updates the competitive doc, the demo writeup table/footnotes, and adds a same-set head-to-head bar to the investor page. Full accounting + reproduction in docs/benchmarks/results-2026-06-15-hhem-head-to-head.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add site/eu-ai-act.html — a European-market page mapping the AI Act's high-risk obligations (Articles 12/13/14/9/72) to the specific artifacts Orc produces, grounded in docs/compliance/eu-ai-act.md. Deployed via Pagenta at https://thor.pagenta.app/orc-eu-ai-act. Keeps the doc's honest framing (Orc produces the evidence; the obligation stays with the deployer) and accurate dates: Aug 2026 broad applicability, Dec 2027 high-risk Annex III obligations. Also fix invest.html, which imprecisely put high-risk obligations at Aug 2026 — corrected to Dec 2027 with the Aug 2026 broad-applicability context, matching the compliance doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the abstract F1 number concrete: run the same model two ways — a bare LLM call vs orc — on real claims, with real captured output. Two demos, one honest writeup.
demos/orc_vs_raw.py— single-passage worked examples (live, gated).demos/orc_large_corpus.py+demos/corpus/— an 11-document fictional private knowledge base; the case a raw call can't handle.docs/demos/defensible-verification.md— captured output, comparison table, citation-invariant result, competitive table, reproduction commands.The four examples (all honest, nothing rigged)
halueval-803(fabricated award) — both catch it; orc adds the defensible artifact. Tie.financebench_id_02747(AMD CAGR) — both correct; orc's math lands in the trace. Tie.financebench_id_07081(CVS ROA) — found by scanning DROP+FinanceBench for raw failures. The production-shaped raw call (quick YES/NO) answers "YES" and rubber-stamps a wrong 0.04 ROA; orc's arithmetic mode computes 0.03 first and rejects. orc-correct, raw-wrong.Honest framing (matches
competitive.md)Two regimes, stated plainly: on short single passages a frontier model is already a strong judge, so orc's value is the defensible record; on hard numeric items the quick raw verdict breaks and orc holds; on a private corpus a raw call can't verify at all. orc has its own misses too (~0.86 F1, not 1.0) — the doc says so. Plus the free, reproducible citation invariant (0/300 fabricated IDs leaked).
Testing
src/orcchanged; demos are not packaged into the wheel/sdist.Not included (flagged in the doc)
--hhem) but not yet run; table cites published numbers.🤖 Generated with Claude Code
Plus: investor/prospect landing page (
site/invest.html→/invest)A standalone page in the existing brand, built around the live proof receipts (CVS ROA correctness gap + private-corpus citation). Responsive, honest claims, noscript fallback. Screenshotted desktop + mobile during build.