Skip to content

Spec B: shard and verify an official ChaosNLI split#13

Merged
AMBRA7592 merged 3 commits into
mainfrom
codex/chaosnli-sharding-part3a
Jul 18, 2026
Merged

Spec B: shard and verify an official ChaosNLI split#13
AMBRA7592 merged 3 commits into
mainfrom
codex/chaosnli-sharding-part3a

Conversation

@AMBRA7592

Copy link
Copy Markdown
Collaborator

Summary

  • add deterministic contiguous shard and offset/limit selection to the ChaosNLI adapter
  • emit deterministic per-shard manifests and aggregate them with an exact ordered-UID coverage proof that rejects missing indices, gaps, overlaps, duplicates, and reordering
  • scale cohort lookup and leave-one-out reliability without changing observable behavior
  • add a Tier-2 harness and reproducibility evidence for the official ChaosNLI v1.0 SNLI split, while committing no licensed rows or converted outputs

Commit structure

  1. 8f40405 — sharding, manifests, synthetic fixture/tests, and report harness
  2. b6b9914 — independently revertible disagreement.py performance refactor plus seeded naive-vs-optimized equivalence test
  3. cf3909a — evidence only: official-run manifest/report and reproduction instructions

The third commit is intentionally evidence-only: the committed manifest names the already-existing exact code commit used for the run (b6b9914). This preserves truthful provenance while keeping the two mandated code changes separate.

Tier 1 — CI-pinned logic

  • 7 synthetic records over 3 uneven shards yield sizes 2/2/3 and exact ordered coverage
  • offset/limit output is byte-identical to the corresponding shard ranges
  • duplicate and gap/order corruptions are rejected after internally consistent manifest mutation
  • broken converted distributions are rejected
  • manifests reproduce byte-for-byte
  • optimized reliability/cohort lookup equals the naive former implementation across forced ties, singleton cells, overlapping cohorts, and 250 seeded randomized configurations

Tier 2 — external empirical evidence

  • official source: chaosNLI_snli.jsonl, ChaosNLI v1.0, CC BY-NC 4.0
  • source SHA-256: 99f9015ddda7d85f66a087452bc30d53974314fe27e7d589e2f41ad44bd509c1
  • 1,514 records / 151,400 anonymous votes / 16 shards
  • distributions: 1,514/1,514 exact
  • soft labels: 1,514/1,514 exact
  • entropies: 1,514/1,514 at documented 3-decimal output precision
  • schema validation and operational pipeline: 16/16 shards
  • second complete run reproduced manifest.json byte-for-byte (1406173c…)
  • reliability, error attribution, variation-vs-error, and cohort-fork prevalence are explicitly NON-APPLICABLE and undefined, not zero

Regression and scope gates

  • default triage.json, soft_labels.jsonl, soft_labels.csv, and governance.jsonl are byte-identical to verified main
  • resolution records are identical after removing runtime timestamps; replay hashes match exactly
  • genuinely jsonschema-free venv: 31 tests, 29 pass + 2 intended skips
  • jsonschema-enabled: 31/31 pass
  • exact tracked scope: .gitignore, adapter code/docs/fixture, disagreement.py, test_claims.py, and reports/chaosnli/
  • no official source row, UID, converted labels file, or virtual-voter record is committed

Audit note

This PR is intentionally left unmerged for independent release-gate review.

@AMBRA7592
AMBRA7592 merged commit 3dd8726 into main Jul 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant