This is the complete, frozen v1 kit behind the published study:
Silent Failures in Single-Shot Text-to-SQL: A Blind, Nine-Model Cross-Comparison on the "Ground Truth" Benchmark — Wang, Huayin (2026). DOI: https://doi.org/10.5281/zenodo.21349581
benchmark/— the benchmark itself, as-frozen: generator (v0.3.0-g4), harness/grader, the CI reference instancedc81f6e0c4182384(300K; warehouse, latent ground truth, 111-question manifest with answer keys, schema docs), release commitments, run guides.study/— the nine-model study kit: the exact model-facing inputs (INSTRUCTIONS.md,questions.md,schema_reference.md), all nine raw answer sets (verbatim), the scoring drivers, the committed comparison artifacts, andFINDINGS.md.tables/comparison.csv— the published comparison tables as data.reproduction/REPRODUCTION_LOG.md— an independent re-scoring of all nine answer sets through the harness's own path, matching the committed tables exactly (2026-07-14).ERRATA.md— the grading/key history, stated plainly.MANIFEST-CHECK.md/GAPS.md— mechanical inventory against the publication manifest; every item is a path or an explicit gap with a reason.
cd benchmark
pip install duckdb pandas
python score_uploaded.py instances/dc81f6e0c4182384 ../study/results/answers_<model>.json --label <model>
Expected: the trust scores and silent counts in tables/comparison.csv, exactly.
v1 is published here in full — questions, ground truth, answer keys, raw model answers —
for reproducibility and audit. It is therefore retired as an evaluation instrument:
once a benchmark's answers are public, future models may absorb them, and scores on it
stop meaning what they meant. Future scored runs use a private revision of the benchmark,
for the obvious reason. (The sealed competition holdout referenced in
benchmark/PICKUP.md remains sealed and is not part of this kit — by design.)
The engine under study was renamed after this benchmark was frozen; artifacts herein use
the historical name ("Coframe" / coframe_def.cf, coframe-benchmark). They are
published as-frozen: renaming inside a frozen research artifact would falsify the record.
The current project is Columna, by datumwise (https://datumwise.ai).
Documents and data: CC-BY-4.0. Code (generator, harness, drivers): Apache-2.0. See LICENSE-NOTICE.md; full license texts accompany the public repository.