Add a single benchmark artifact verification path

.github/workflows/ci.yml

@@ -35,22 +35,4 @@ jobs:
             --output-dir /tmp/l20_codeforge_scorecard_ci
 
       - name: Verify benchmark artifact hashes
-        run: |
-          python - <<'PY'
-          import hashlib
-          from pathlib import Path
-
-          expected = {
-              "benchmarks/generalization_scorecard_2026_05_23/scorecard.json":
-                  "1eb0402378ea25732225b29d7ba367b6111ab3351e54cc7c01fa7646a7a12712",
-              "benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json":
-                  "2a0ff919aa15eb9ecdf74824f7bf790a23f6d0197ef74970b6190c60e0e00772",
-              "benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv":
-                  "08732bbb76450f92ef3c02fa97a163aba01f71028365072c205c5a3af45d5550",
-          }
-          for rel_path, expected_sha in expected.items():
-              actual_sha = hashlib.sha256(Path(rel_path).read_bytes()).hexdigest()
-              if actual_sha != expected_sha:
-                  raise SystemExit(f"{rel_path}: expected {expected_sha}, got {actual_sha}")
-          print("benchmark artifact hashes verified")
-          PY
+        run: python -m l20_codeforge verify-artifacts

README.md

@@ -1,6 +1,6 @@
 # L20-CodeForge
 
-[![CI](https://github.com/Kevin-Li-2025/L20-CodeForge/actions/workflows/ci.yml/badge.svg)](https://github.com/Kevin-Li-2025/L20-CodeForge/actions/workflows/ci.yml)
+[![CI](https://github.com/yinli-systems/L20-CodeForge/actions/workflows/ci.yml/badge.svg)](https://github.com/yinli-systems/L20-CodeForge/actions/workflows/ci.yml)
 
 Single-L20 post-training, verifier-guided inference, and executable benchmark
 infrastructure for code models.
@@ -13,9 +13,9 @@ generation, repair, verifier-guided inference, trajectory data, and reward
 signals for code models.
 
 For serving, kernel, and runtime infrastructure work, use
-[l20-stack](https://github.com/Kevin-Li-2025/l20-stack). For from-scratch
+[l20-stack](https://github.com/yinli-systems/l20-stack). For from-scratch
 pretraining and public checkpoint release artifacts, use
-[l20-edu-135m-pretrain](https://github.com/Kevin-Li-2025/l20-edu-135m-pretrain).
+[l20-edu-135m-pretrain](https://github.com/yinli-systems/l20-edu-135m-pretrain).
 This repository should stay focused on executable coding benchmarks rather than
 becoming a second general L20 infrastructure repo.
 
@@ -99,15 +99,14 @@ python3 -m venv .venv
 source .venv/bin/activate
 python -m pip install -e ".[dev,bench]"
 python -m pytest -q
+python -m l20_codeforge verify-artifacts
 python -m l20_codeforge profile
 python -m l20_codeforge smoke-loop
 ```
 
-The `python -m pytest -q` line should print:
-
-```text
-135 passed in <time>s
-```
+The test command should finish with no failures. `verify-artifacts` should return
+`"status": "PASS"`; it checks every claim-bearing file listed in
+`REPRODUCIBILITY.md`, rather than duplicating a shorter hash list in CI.
 
 On an L20 host:
 

REPRODUCIBILITY.md

@@ -39,11 +39,11 @@ python -m pytest -q
 Expected output shape:
 
 ```text
-135 passed in <time>s
+<count> passed in <time>s
 ```
 
-The exact wall time depends on machine load. The pass count is the important
-invariant for this snapshot.
+The exact wall time and collected count can change as regression coverage grows;
+zero failures is the invariant.
 
 ## Static L20 Profile
 
@@ -95,33 +95,34 @@ check must have `"passed": true`.
 
 ## Benchmark Artifact Hashes
 
-Verify the committed artifact hashes:
+Verify every committed artifact used by the headline result tables:
 
 ```bash
-shasum -a 256 \
-  benchmarks/generalization_scorecard_2026_05_23/scorecard.json \
-  benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json \
-  benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv \
-  benchmarks/livecodebench_full_release_v6_2026_05_22/qwen25_coder_7b_temp08_n8_public_select_full_eval/report.json \
-  benchmarks/evalplus_l20_codeforge_2026_05_22/rechecks/manifest.json
+python -m l20_codeforge verify-artifacts
 ```
 
-Expected output:
+Expected output begins with:
 
-```text
-1eb0402378ea25732225b29d7ba367b6111ab3351e54cc7c01fa7646a7a12712  benchmarks/generalization_scorecard_2026_05_23/scorecard.json
-2a0ff919aa15eb9ecdf74824f7bf790a23f6d0197ef74970b6190c60e0e00772  benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json
-08732bbb76450f92ef3c02fa97a163aba01f71028365072c205c5a3af45d5550  benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv
-7272f5591c2f868c059226a2a5ec8fc772994cfafd20eb8397a2b6d90aed64bf  benchmarks/livecodebench_full_release_v6_2026_05_22/qwen25_coder_7b_temp08_n8_public_select_full_eval/report.json
-e86db2af864a9c8896dcd1bc2d4d7b44af7fa395b856ea02b6f0e69c31c915cc  benchmarks/evalplus_l20_codeforge_2026_05_22/rechecks/manifest.json
+```json
+{
+  "status": "PASS",
+  "artifacts": [
+    {
+      "path": "benchmarks/generalization_scorecard_2026_05_23/scorecard.json",
+      "status": "ok"
+    }
+  ]
+}
 ```
 
-The CI workflow verifies the three top-level claim artifacts on every push and
-pull request:
+The full output includes expected and actual SHA-256 values. CI checks these five
+claim artifacts on every push and pull request:
 
 - `benchmarks/generalization_scorecard_2026_05_23/scorecard.json`
 - `benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json`
 - `benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv`
+- `benchmarks/livecodebench_full_release_v6_2026_05_22/qwen25_coder_7b_temp08_n8_public_select_full_eval/report.json`
+- `benchmarks/evalplus_l20_codeforge_2026_05_22/rechecks/manifest.json`
 
 ## LiveCodeBench Full Replay Boundary
 

src/l20_codeforge/artifacts.py

@@ -0,0 +1,80 @@
+from __future__ import annotations
+
+import hashlib
+from collections.abc import Mapping
+from dataclasses import asdict, dataclass
+from pathlib import Path
+
+DEFAULT_ARTIFACT_HASHES: dict[str, str] = {
+    "benchmarks/generalization_scorecard_2026_05_23/scorecard.json": (
+        "1eb0402378ea25732225b29d7ba367b6111ab3351e54cc7c01fa7646a7a12712"
+    ),
+    "benchmarks/livecodebench_full_release_v6_2026_05_22/"
+    "full_n8_public_select_summary.json": (
+        "2a0ff919aa15eb9ecdf74824f7bf790a23f6d0197ef74970b6190c60e0e00772"
+    ),
+    "benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv": (
+        "08732bbb76450f92ef3c02fa97a163aba01f71028365072c205c5a3af45d5550"
+    ),
+    "benchmarks/livecodebench_full_release_v6_2026_05_22/"
+    "qwen25_coder_7b_temp08_n8_public_select_full_eval/report.json": (
+        "7272f5591c2f868c059226a2a5ec8fc772994cfafd20eb8397a2b6d90aed64bf"
+    ),
+    "benchmarks/evalplus_l20_codeforge_2026_05_22/rechecks/manifest.json": (
+        "e86db2af864a9c8896dcd1bc2d4d7b44af7fa395b856ea02b6f0e69c31c915cc"
+    ),
+}
+
+
+@dataclass(frozen=True)
+class ArtifactVerification:
+    path: str
+    expected_sha256: str
+    actual_sha256: str | None
+    status: str
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def verify_artifacts(
+    root: Path = Path("."),
+    expected_hashes: Mapping[str, str] = DEFAULT_ARTIFACT_HASHES,
+) -> list[ArtifactVerification]:
+    """Verify claim-bearing artifacts without reading outside ``root``."""
+    resolved_root = root.resolve()
+    results: list[ArtifactVerification] = []
+    for relative_path, expected_sha256 in expected_hashes.items():
+        candidate = (resolved_root / relative_path).resolve()
+        try:
+            candidate.relative_to(resolved_root)
+        except ValueError:
+            raise ValueError(f"artifact path escapes root: {relative_path}") from None
+
+        if not candidate.is_file():
+            results.append(
+                ArtifactVerification(relative_path, expected_sha256, None, "missing")
+            )
+            continue
+        actual_sha256 = sha256_file(candidate)
+        status = "ok" if actual_sha256 == expected_sha256 else "mismatch"
+        results.append(
+            ArtifactVerification(relative_path, expected_sha256, actual_sha256, status)
+        )
+    return results
+
+
+def artifact_report(
+    root: Path = Path("."),
+    expected_hashes: Mapping[str, str] = DEFAULT_ARTIFACT_HASHES,
+) -> dict[str, object]:
+    results = verify_artifacts(root=root, expected_hashes=expected_hashes)
+    return {
+        "status": "PASS" if all(result.status == "ok" for result in results) else "FAIL",
+        "artifacts": [asdict(result) for result in results],
+    }

src/l20_codeforge/cli.py

@@ -7,6 +7,7 @@
 from rich.console import Console
 
 from l20_codeforge.agents.mini_swe import convert_mini_trajectory_file, export_mini_task_records
+from l20_codeforge.artifacts import artifact_report
 from l20_codeforge.context.compiler import ContextCompiler
 from l20_codeforge.data.code_bench_sft import build_mbpp_sft_jsonl
 from l20_codeforge.data.preferences import build_preference_pairs
@@ -21,8 +22,8 @@
     generate_evalplus_samples,
     run_evalplus_official,
     select_evalplus_by_base_tests,
-    select_evalplus_by_public_consensus,
     select_evalplus_by_prompt_doctests,
+    select_evalplus_by_public_consensus,
 )
 from l20_codeforge.evals.patch_eval import evaluate_patch, load_task
 from l20_codeforge.evals.real_exec import evaluate_real_patch
@@ -35,6 +36,19 @@
 console = Console()
 
 
+@app.command("verify-artifacts")
+def verify_artifacts_command(root: Path = Path(".")) -> None:
+    """Verify hashes for the committed benchmark evidence used by the README."""
+    try:
+        report = artifact_report(root)
+    except ValueError as exc:
+        console.print(f"[red]failed to verify artifacts:[/red] {exc}")
+        raise typer.Exit(1) from exc
+    console.print_json(data=report)
+    if report["status"] != "PASS":
+        raise typer.Exit(1)
+
+
 @app.command()
 def profile() -> None:
     """Print the static L20 memory policy."""

tests/test_artifacts.py

@@ -0,0 +1,35 @@
+from __future__ import annotations
+
+import hashlib
+from pathlib import Path
+
+import pytest
+
+from l20_codeforge.artifacts import artifact_report, verify_artifacts
+
+
+def test_artifact_report_distinguishes_ok_missing_and_mismatch(tmp_path: Path) -> None:
+    good = tmp_path / "good.json"
+    good.write_bytes(b"stable artifact\n")
+    bad = tmp_path / "bad.json"
+    bad.write_bytes(b"changed artifact\n")
+    expected = {
+        "good.json": hashlib.sha256(good.read_bytes()).hexdigest(),
+        "bad.json": hashlib.sha256(b"original artifact\n").hexdigest(),
+        "missing.json": hashlib.sha256(b"missing\n").hexdigest(),
+    }
+
+    report = artifact_report(tmp_path, expected)
+
+    assert report["status"] == "FAIL"
+    statuses = {item["path"]: item["status"] for item in report["artifacts"]}
+    assert statuses == {
+        "good.json": "ok",
+        "bad.json": "mismatch",
+        "missing.json": "missing",
+    }
+
+
+def test_artifact_paths_cannot_escape_repository_root(tmp_path: Path) -> None:
+    with pytest.raises(ValueError, match="escapes root"):
+        verify_artifacts(tmp_path, {"../outside.json": "0" * 64})