Harden benchmark generator contract with mandatory splits.yaml and split-aware dataset layout

[Mtrya]

Summary

Treat benchmark dataset generation as a first-class, reproducible contract by mandating splits.yaml for all finished benchmarks and unifying generator entrypoints around it. Dataset cases are organized under dataset/cases/<split>/<case_id>/, even for benchmarks that only define a single split (e.g. test).

Why

• Parameters that control dataset construction are currently hardcoded or scattered across ad-hoc CLI flags; readers must reverse-engineer generator code to understand how cases are built.
• A single, mandatory splits.yaml surfaces those parameters explicitly and makes extending or modifying a dataset an editing task rather than a code task.
• A consistent split layout prevents ad-hoc per-benchmark conventions from diverging.
• Generators that are driven by YAML can reproduce the full dataset, a single split, or a custom split without requiring manual reorganization.

Scope

• Define the split layout convention: dataset/cases/<split>/<case_id>/ (replaces the flat dataset/cases/<case_id>/ for finished benchmarks).
• Define the mandatory splits.yaml location: benchmarks/<name>/splits.yaml (committed, canonical).
• Specify the generator CLI contract:
  • Entrypoint: python benchmarks/<name>/generator.py path/to/splits.yaml
  • Running without arguments is an error (prints usage).
  • All dataset-construction arguments move into splits.yaml; operational flags like --help may remain as optional CLI behavior.
• Specify two supported splits.yaml schemas:
  1. Split parameters — for algorithmic generators that build cases per split:

     splits:
       easy:
         seed: 42
         case_count: 5
         max_satellites: 3
       hard:
         seed: 142
         case_count: 5
         max_satellites: 12

  2. Split assignments — for fixed-case benchmarks that bucket existing cases:

     splits:
       test:
         - case_001
         - case_002
       train:
         - case_003
         - case_004

• Update dataset/index.json:
  • Add optional example_smoke_case field containing a relative path: "test/case_0001".
• Update the benchmark template and docs/benchmark_contract.md to cover the new contract.
• Update scripts/validate_benchmark_contract.py:
  • Validate split directories match splits.yaml.
  • Update smoke-test case resolution.
  • Enforce the new generator CLI shape.
• Apply the new contract to all benchmark generators

Out Of Scope

• Changing the verifier interface — verifiers operate on individual case directories and do not need to be split-aware.
• Solution code or baselines.
• Regenerating cases — existing cases should be manually git mved to new directories to ensure cases don't drift.
• Updating i18n docs (will be handled in issue CI: Add i18n Documentation Sync Check #27 .

Acceptance Criteria

• splits.yaml is mandatory for every finished benchmark; generators fail when invoked without a YAML path.
• Generator entrypoint consistently takes a splits.yaml path as its required argument.
• Split layout (dataset/cases/<split>/<case_id>/) is documented as the canonical dataset layout.
• Finished benchmarks have migrated to the new contract.
• scripts/validate_benchmark_contract.py enforces the new rules.
• The verifier contract is unchanged.
• Flat layout is removed

Notes

• Splits are a dataset organization and generator-contract convention. The verifier should remain fully agnostic to whether cases come from a split subdirectory or the flat root.
• The reference implementation should be chosen from a benchmark whose generator is already stable.
• A shared top-level shape like splits: makes sense across benchmarks, but the exact per-split parameters should remain benchmark-owned where needed, rather than forcing one universal schema across all generators.

[Mtrya]

Proposal for hardening issue #19

After offline discussion, here is a design proposal aims at hardening this issue.

---

1. Layout convention

Mode	Directory structure
Flat (current)	dataset/cases/<case_id>/
Split (target)	dataset/cases/<split>/<case_id>/

2. Split names

• Arbitrary strings are allowed.
• CI validates splits in splits.yaml

3. YAML config (splits.yaml)

Placed at benchmark top level: benchmarks/<name>/splits.yaml (canonical, committed). Two supported schemas:

A. Split parameters — for algorithmic generators that build cases per split:

splits:
  easy:
    seed: 42
    case_count: 5
    max_satellites: 3
  hard:
    seed: 142
    case_count: 5
    max_satellites: 12

B. Split assignments — for fixed-case benchmarks that bucket existing cases:

splits:
  test:
    - case_001
    - case_002
  train:
    - case_003
    - case_004

Single-split YAML is valid (e.g., only test).

4. Generator CLI (breaking change)

• Entrypoint becomes python benchmarks/<name>/generator.py path/to/splits.yaml
• Or python -m benchmarks.<name>.generator.run path/to/splits.yaml
• All other CLI flags removed. Every generator argument moves into the YAML.
• Running without arguments is an error (prints usage).
• Each benchmark commits a canonical YAML that reproduces the splits.

5. Verifier contract

• Unchanged. Verifiers remain split-agnostic.
• Accepts split paths directly: verifier.py dataset/cases/train/case_001 solution.json
• dataset/index.json gains optional example_smoke_case_split field for CI smoke tests.

6. Documentation & CI updates needed

• docs/benchmark_contract.md: new dataset layout rules, generator entrypoint policy, YAML schemas, index.json fields.
• scripts/validate_benchmark_contract.py: detect split mode, validate split directories, update smoke-test case resolution, enforce new generator CLI shape.
• GitHub Actions reproducibility workflow: split-aware if necessary.

[Mtrya]

A clarification on the direction here, based on the offline discussion:

I support treating this as a generator-contract cleanup, not only a dataset-layout tweak.

A few points that seem worth making explicit:

1. splits.yaml should be mandatory for finished benchmarks, and generators should not still work with no args.

   • The no-arg path should fail.
   • Passing an explicit YAML path should be the standard way.

2. The main value of splits.yaml is not only split naming, but surfacing generator parameters that are currently hardcoded or scattered across CLI flags.

   • Readers should be able to understand the intended dataset construction by reading the YAML, not reverse-engineering generator code.
   • Creating more cases or changing difficulty/scaling should mostly mean editing YAML, not editing Python.

3. I would keep the split layout under dataset/cases/, i.e. dataset/cases/<split>/<case_id>/.

   • That preserves the current dataset-root contract around cases/, index.json, and example_solution.json.
   • Verifiers can remain fully split-agnostic and continue accepting a case directory path.

4. I would avoid introducing two public generator contracts.

   • If we decide YAML is the canonical interface, then all finished benchmarks should adopt it, even if many only define a single test split.
   • That is clearer than supporting a mixed “some benchmarks have YAML, some do not” model.

5. I would still leave room for benchmark-specific config fields inside a common outer structure.

   • A shared top-level shape like splits: makes sense.
   • But the exact per-split parameters should remain benchmark-owned where needed, rather than forcing one universal schema across all generators.

6. For smoke tests, I think a single relative path field would be simpler than splitting case id and split into separate metadata.

   • For example: example_smoke_case: "test/case_0001" in split mode, or "case_0001" in flat mode.

7. I would not move every runtime concern into YAML.

   • Dataset-construction parameters belong in YAML.
   • Operational concerns like --help, and possibly things like --force-download, can stay as optional CLI behavior.

So my take is: yes to mandatory splits.yaml, yes to using it to expose generator parameters, and yes to treating this as an intentional repo-wide public contract change rather than a small optional extension.