Skip to content

Add a single benchmark artifact verification path - #1

Open
yinli-systems wants to merge 1 commit into
mainfrom
agent/add-artifact-verifier
Open

Add a single benchmark artifact verification path#1
yinli-systems wants to merge 1 commit into
mainfrom
agent/add-artifact-verifier

Conversation

@yinli-systems

Copy link
Copy Markdown
Owner

What changed

  • add l20-codeforge verify-artifacts with structured PASS/FAIL results, missing/mismatch reporting, and repository-root containment
  • verify all five claim-bearing artifacts documented in REPRODUCIBILITY.md
  • replace the shorter inline CI hash script with the same package command users run locally
  • add regression tests for valid, changed, missing, and escaping artifact paths
  • update canonical repository links and make README/test expectations resistant to normal test-count growth

Why

The reproducibility guide listed five hashes, but CI checked only three using a duplicated inline map. That allowed evidence drift between the documented and automated verification paths.

Impact

README claims, local verification, and CI now share one manifest and one implementation. The benchmark numbers and artifacts are unchanged.

Checks

  • PATH="$PWD/.venv/bin:$PATH" python -m pytest -q — 137 passed
  • python -m l20_codeforge verify-artifacts — PASS, 5/5 hashes matched
  • python scripts/build_generalization_scorecard.py --output-dir <temp> — PASS, all 9 checks passed
  • ruff check src/l20_codeforge/artifacts.py src/l20_codeforge/cli.py tests/test_artifacts.py — passed
  • git diff --check — passed

Full hidden LiveCodeBench replay and GPU/model execution remain outside this documentation/verification change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant