docs: raw-LLM-vs-orc demos (worked examples + large private corpus)

README.md

@@ -183,6 +183,8 @@ What shipped in v0.2.0:
 
 Shipped earlier in v0.1.4: `--include-evidence` self-contained audit bundles, `mode="arithmetic"` with a safe AST-walking calculator (FinanceBench F1 0.736 → 0.916), the evidence-mode citation guard, and model-agnostic self-hosting of any open-weight judge.
 
+Raw LLM vs. orc, side by side on real claims — including a large private corpus the model has never seen, where a bare call can't verify at all: **[docs/demos/defensible-verification.md](./docs/demos/defensible-verification.md)**. Run them yourself: `uv run python -m demos.orc_vs_raw --live` and `uv run python -m demos.orc_large_corpus --live`.
+
 Live walkthrough: **[pagenta.app/p/thorm/orc-how-it-works](https://pagenta.app/p/thorm/orc-how-it-works)** — six-scene visual explainer. Full pitch: **[pagenta.app/p/thorm/orc-pitch](https://pagenta.app/p/thorm/orc-pitch)**.
 
 Full per-source breakdown + reproducing instructions: [`docs/benchmarks/results-2026-05-19-phase2-arithmetic.md`](./docs/benchmarks/results-2026-05-19-phase2-arithmetic.md). Multi-model portability (Sonnet, Haiku, GPT-4o, Gemini Flash, Llama 3.3 70B): [`docs/benchmarks/results-2026-05-19-multi-model.md`](./docs/benchmarks/results-2026-05-19-multi-model.md). Competitive positioning: [`docs/positioning/competitive.md`](./docs/positioning/competitive.md). EU AI Act mapping: [`docs/compliance/eu-ai-act.md`](./docs/compliance/eu-ai-act.md). Business model + stage-by-stage roadmap: [`docs/business/roadmap.md`](./docs/business/roadmap.md). Cost economics across all tested models (Sonnet, Haiku, GPT-4o, Gemini Flash, Llama, Qwen, Gemma): [`docs/business/cost-economics.md`](./docs/business/cost-economics.md).

benchmarks/faithfulness/head_to_head.py

@@ -0,0 +1,156 @@
+"""Same-set head-to-head: Orc vs. Vectara HHEM-2.1-Open on identical items.
+
+Unlike `run.py --hhem` (which re-runs Orc and HHEM together, spending live LLM
+budget on the Orc pass), this script *reuses* an existing Orc results.json and
+only runs the (free, self-hosted) HHEM pass on the exact same items, matched by
+id. That makes the comparison genuinely same-set and costs nothing on the Orc
+side.
+
+HHEM-2.1-Open ships vendored modeling code written for transformers 4.x; it
+breaks on transformers 5.x (`all_tied_weights_keys`). The `benchmarks` extra
+pins `transformers<5` for this reason — that is the fix for the long-standing
+"HHEM tokenizer-load issue."
+
+    uv run python -m benchmarks.faithfulness.head_to_head \
+        --orc-run benchmarks/faithfulness/results/20260519-191250/results.json
+
+Output: a same-set table (Orc vs HHEM), a per-source breakdown, and a
+truncation split (HHEM has a 512-token window) so a reviewer can see how much
+of the gap — if any — is attributable to HHEM not seeing long passages.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+import warnings
+from collections import defaultdict
+from pathlib import Path
+from typing import Any
+
+warnings.filterwarnings("ignore")
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+DATASET_PATH = Path(__file__).parent / "halubench-stratified-504.jsonl"
+THRESHOLD = 0.5  # HHEM P(consistent) >= THRESHOLD -> PASS (faithful)
+TOKENS_PER_WORD = 1.3  # rough proxy for HHEM's 512-token window
+
+
+def _load(orc_run: Path) -> tuple[list[dict[str, Any]], dict[str, dict[str, Any]]]:
+    ds = {json.loads(line)["id"]: json.loads(line) for line in DATASET_PATH.open()}
+    orc_items = [
+        it
+        for it in json.loads(orc_run.read_text())["items"]
+        if it.get("orc_binary") and it.get("id") in ds
+    ]
+    return orc_items, ds
+
+
+def _score(rows: list[dict[str, Any]], key: str) -> dict[str, float]:
+    from orc.metrics.scoring import LabeledResult, confusion, scores
+
+    cm = confusion(
+        [LabeledResult(predicted=r[key], expected=r["gt"]) for r in rows], positive="PASS"
+    )
+    return {**scores(cm), **{f"cm_{k}": v for k, v in cm.items()}}
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--orc-run",
+        type=Path,
+        required=True,
+        help="Path to an existing Orc faithfulness results.json to reuse verdicts from",
+    )
+    parser.add_argument("--threads", type=int, default=10, help="torch CPU threads")
+    parser.add_argument("--out", type=Path, default=None, help="Write results JSON here")
+    args = parser.parse_args(argv)
+
+    sys.path.insert(0, str(REPO_ROOT / "src"))
+    import torch
+
+    torch.set_num_threads(args.threads)
+    from transformers import AutoModelForSequenceClassification
+
+    orc_items, ds = _load(args.orc_run)
+    print(f"reusing {len(orc_items)} Orc verdicts from {args.orc_run.name}")
+
+    rows: list[dict[str, Any]] = []
+    for it in orc_items:
+        d = ds[it["id"]]
+        words = len((d["passage"] + " " + d["answer"]).split())
+        rows.append(
+            {
+                "id": it["id"],
+                "source_ds": it["source_ds"],
+                "gt": it["ground_truth"],
+                "orc": it["orc_binary"],
+                "passage": d["passage"],
+                "answer": d["answer"],
+                "est_tok": int(words * TOKENS_PER_WORD),
+            }
+        )
+
+    print("loading HHEM-2.1-Open (self-hosted, free)…")
+    model = AutoModelForSequenceClassification.from_pretrained(
+        "vectara/hallucination_evaluation_model", trust_remote_code=True
+    )
+    model.eval()
+
+    print(f"scoring {len(rows)} items (premise=passage, hypothesis=answer, threshold={THRESHOLD})…")
+    t0 = time.monotonic()
+    batch = 8
+    for i in range(0, len(rows), batch):
+        chunk = rows[i : i + batch]
+        scores_out = model.predict([(r["passage"], r["answer"]) for r in chunk])
+        for r, s in zip(chunk, scores_out, strict=True):
+            r["hhem_score"] = float(s)
+            r["hhem"] = "PASS" if float(s) >= THRESHOLD else "FAIL"
+        if (i // batch) % 10 == 0:
+            print(f"  {min(i + batch, len(rows))}/{len(rows)}")
+    print(f"  done in {time.monotonic() - t0:.0f}s\n")
+
+    orc_s, hhem_s = _score(rows, "orc"), _score(rows, "hhem")
+    print("=" * 70)
+    print(f"SAME-SET HEAD-TO-HEAD  ·  n={len(rows)}  ·  PASS = faithful")
+    print("=" * 70)
+    for name, s in (("orc", orc_s), ("hhem", hhem_s)):
+        print(
+            f"  {name:5} F1={s['f1']:.4f}  acc={s['accuracy']:.4f}  "
+            f"P={s['precision']:.4f}  R={s['recall']:.4f}"
+        )
+
+    fits = [r for r in rows if r["est_tok"] <= 512]
+    trunc = [r for r in rows if r["est_tok"] > 512]
+    print(f"\ntruncation split (HHEM 512-token window; ~{len(fits)}/{len(rows)} fit):")
+    for label, sub in (("fits<=512", fits), ("truncated", trunc)):
+        if sub:
+            print(
+                f"  {label:10} n={len(sub):3}  orc F1={_score(sub,'orc')['f1']:.3f}  "
+                f"hhem F1={_score(sub,'hhem')['f1']:.3f}"
+            )
+
+    print("\nper source_ds (F1):")
+    by: dict[str, list] = defaultdict(list)
+    for r in rows:
+        by[r["source_ds"]].append(r)
+    for src in sorted(by):
+        rs = by[src]
+        print(
+            f"  {src:13} n={len(rs):3}  orc={_score(rs,'orc')['f1']:.3f}  "
+            f"hhem={_score(rs,'hhem')['f1']:.3f}"
+        )
+
+    if args.out:
+        args.out.write_text(
+            json.dumps({"orc": orc_s, "hhem": hhem_s, "n": len(rows), "rows": rows}, indent=2)
+        )
+        print(f"\nwrote {args.out}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

demos/corpus/architecture.md

@@ -0,0 +1,8 @@
+# Platform Architecture
+
+The Helix platform is a set of Python services behind an API gateway. The
+Checkout API and Tracking Mesh each own a PostgreSQL 15 database; cross-service
+reads go through a read-replica. Async work runs on a Redis-backed queue.
+Services are deployed as containers on a managed Kubernetes cluster in eu-north-1
+with a warm standby in eu-central-1. Connection pooling to PostgreSQL is handled
+per-service with a configurable max-connections cap.

demos/corpus/company-overview.md

@@ -0,0 +1,8 @@
+# Helix Freight Systems — Company Overview
+
+Helix Freight Systems is a (fictional) logistics-software company founded in
+2019, headquartered in Tallinn. We operate a freight-booking and tracking
+platform used by mid-market carriers across the EU. As of 2026 we have 240
+employees and serve roughly 1,800 carrier accounts. Our flagship products are
+the Checkout API (carrier booking), the Tracking Mesh (shipment telemetry), and
+the Helix Console (operator dashboard).

demos/corpus/data-retention-policy.md

@@ -0,0 +1,7 @@
+# Data Retention Policy
+
+Shipment telemetry is retained for 18 months, then aggregated and the raw rows
+deleted. Booking records are retained for 7 years to meet carrier audit
+requirements. Application logs are retained 90 days hot, 1 year cold. Customers
+on the Enterprise tier may request a custom retention window. Deletion requests
+under GDPR are honored within 30 days.

demos/corpus/glossary.md

@@ -0,0 +1,9 @@
+# Glossary
+
+- **Booking**: a single freight reservation created via the Checkout API.
+- **Tracking Mesh**: the telemetry pipeline that ingests shipment status events.
+- **Pool utilization**: the fraction of a service's database connection pool in
+  use; the primary leading indicator for checkout latency.
+- **Circuit breaker**: a load-shedding mechanism that rejects excess requests to
+  protect the database during a spike.
+- **SEV-1**: highest-severity incident; customer-facing outage.

demos/corpus/incident-2026-03-14-postmortem.md

@@ -0,0 +1,30 @@
+# Incident Postmortem — 2026-03-14 Checkout Outage
+
+**Status:** Resolved · **Severity:** SEV-1 · **Author:** Priya Nadkarni (SRE)
+
+## Summary
+
+On 2026-03-14, Helix Freight's checkout API was unavailable or degraded for
+**73 minutes**, from 14:02 to 15:15 UTC. Approximately 31,000 checkout
+attempts failed. No data was lost.
+
+## Root cause
+
+The root cause was **database connection-pool exhaustion**. The checkout
+service's pool was capped at `max_connections = 200`. A marketing email sent at
+14:00 UTC drove a 6x traffic spike; the pool saturated within 90 seconds and new
+requests blocked on connection acquisition until they timed out.
+
+This was **not** a code deploy, a TLS/certificate problem, or an upstream
+provider outage — all three were ruled out during triage (see timeline).
+
+## Resolution
+
+1. Raised `max_connections` from 200 to **800** on the checkout pool.
+2. Added a circuit breaker that sheds load at 90% pool utilization.
+3. Capped marketing-email send rate to 2,000 messages/minute.
+
+## Follow-up actions
+
+- HELIX-4471: load-test the pool ceiling before the next campaign (owner: Priya).
+- HELIX-4472: alert on pool utilization > 75% (owner: Marcus).

demos/corpus/onboarding-guide.md

@@ -0,0 +1,7 @@
+# Carrier Onboarding Guide
+
+New carriers complete KYC, then receive sandbox credentials valid for 30 days.
+Integration starts with the Checkout API: create a booking, attach shipment
+metadata, and poll the Tracking Mesh for status. Production credentials are
+issued after a successful sandbox booking and a signed order form. Most carriers
+go live within two weeks. The integration team holds office hours twice weekly.

demos/corpus/oncall-runbook.md

@@ -0,0 +1,8 @@
+# On-Call Runbook
+
+The primary on-call engineer acknowledges pages within 5 minutes. For checkout
+latency alerts: check pool utilization on the Checkout dashboard first, then
+upstream PostgreSQL CPU. To shed load, enable the circuit breaker via the
+feature flag `checkout.circuit_breaker`. Escalate to the database on-call if
+replica lag exceeds 30 seconds. Always open an incident channel for SEV-1 and
+SEV-2 events.

demos/corpus/pricing-tiers.md

@@ -0,0 +1,7 @@
+# Pricing Tiers
+
+Helix offers three tiers. Starter: €0.40 per booking, no SLA, community support.
+Business: €0.28 per booking, 99.9% SLA, 9-to-5 support. Enterprise: custom
+per-booking pricing, 99.9% SLA with credits, 24/7 support, and a dedicated
+solutions engineer. Annual contracts receive a 12% discount. Overage on the
+Tracking Mesh is billed at €0.002 per telemetry event.

demos/corpus/release-notes-v4.md

@@ -0,0 +1,7 @@
+# Release Notes — Checkout API v4.0
+
+v4.0 introduces idempotency keys on all booking endpoints, a 30% reduction in
+p99 latency from query batching, and configurable connection-pool sizing per
+environment. Breaking change: the deprecated `/book/legacy` endpoint is removed.
+v4.0 shipped 2026-02-10. The circuit-breaker feature flag was added later, in a
+2026-03-14 hotfix.

demos/corpus/security-policy.md

@@ -0,0 +1,7 @@
+# Security Policy
+
+All production access requires SSO with hardware-key MFA. Secrets are stored in
+the managed KMS; no secret may be committed to a repository. TLS 1.3 terminates
+at the gateway, and certificates are rotated automatically every 60 days via the
+ACME integration. Access to customer data is least-privilege and reviewed
+quarterly. Penetration tests run twice a year.

demos/corpus/sla.md

@@ -0,0 +1,7 @@
+# Service Level Agreement
+
+Helix commits to 99.9% monthly availability for the Checkout API on the Business
+and Enterprise tiers. Availability is measured at the gateway, excluding
+scheduled maintenance announced 72 hours in advance. SEV-1 incidents trigger
+service credits: 10% of monthly fees for 99.0–99.9% availability, 25% below
+99.0%. The Tracking Mesh carries a separate 99.5% target.