Parent
Status
CLOSEABLE BENCHMARK EPIC — GOVERNED EVALUATION PACKET — NON-CANONICAL UNTIL VALIDATED
Objective
Build a benchmark that tests whether agents and retrieval systems choose the right source families, resolve the right records and versions, preserve exact provenance, find contradictions, and synthesize evidence without flattening source authority.
Benchmark paths
Scientific claim
media or web lead → scholarly graph → paper → dataset → code → execution receipt
Historical claim
newspaper → book → census or government record → archive document → local institutional collection
Technical standard
RFC/NIST/W3C requirement → implementation → issue/PR → package release → conformance test
Company claim
SEC filing → patent → repository → publication → public statement → government or market data
Internal decision
authorized connector evidence → document or discussion → GitHub issue → implementation → public-safe derivative
Required task classes
- exact-record retrieval;
- ambiguous-title and entity resolution;
- current versus historical version selection;
- primary-source recovery from secondary leads;
- contradiction and null-evidence discovery;
- retraction or correction awareness;
- archive/live-page distinction;
- mutable repository reference resolution;
- dataset lineage and mirror detection;
- multimedia claim corroboration;
- private/public boundary preservation;
- incomplete, rate-limited, or inaccessible-source handling.
Required deliverables
- benchmark task and fixture specification;
- source-authority gold or review set;
- route-decision trace;
- exact citation and locator evaluation;
- provenance-completeness checks;
- contradiction and temporal-correctness scoring;
- cost, latency, and failure telemetry where available;
- adversarial cases for fabricated citations, OCR corruption, transcript errors, stale standards, duplicate papers, rehosted datasets, and private-data leakage;
- baseline comparisons across available models and retrieval routes;
- deterministic run receipts and claim-honesty report.
Acceptance criteria
- Every task identifies the expected authority hierarchy and acceptable alternatives.
- Scores separate retrieval success, identity correctness, evidence quality, synthesis correctness, and provenance completeness.
- A cheap route cannot win by omitting required sources or governance gates.
- Archived or stale evidence cannot score as current without explicit temporal posture.
- Fabricated citations and unsupported completeness claims are hard failures.
- At least one task spans four or more source families.
- Results can inform
model-routing-as-code without canonizing one provider or model permanently.
Dependencies
Non-goals
- measuring model trivia recall detached from evidence;
- rewarding verbosity or source count alone;
- treating one static benchmark as permanent proof of route quality.
Parent
Status
CLOSEABLE BENCHMARK EPIC — GOVERNED EVALUATION PACKET — NON-CANONICAL UNTIL VALIDATED
Objective
Build a benchmark that tests whether agents and retrieval systems choose the right source families, resolve the right records and versions, preserve exact provenance, find contradictions, and synthesize evidence without flattening source authority.
Benchmark paths
Scientific claim
media or web lead → scholarly graph → paper → dataset → code → execution receipt
Historical claim
newspaper → book → census or government record → archive document → local institutional collection
Technical standard
RFC/NIST/W3C requirement → implementation → issue/PR → package release → conformance test
Company claim
SEC filing → patent → repository → publication → public statement → government or market data
Internal decision
authorized connector evidence → document or discussion → GitHub issue → implementation → public-safe derivative
Required task classes
Required deliverables
Acceptance criteria
model-routing-as-codewithout canonizing one provider or model permanently.Dependencies
Non-goals