feat: built-in benchmark suite — 4 frozen prompt cards + run/track harness#58
Merged
Conversation
…rness
Adds benchmarks/: four standard prompts (assistant easy/medium/hard +
teacher) with 100-point rubrics split machine-checkable vs judged, a
run-record schema (meta.yaml/transcript/score/artifacts) committed per
run, and autoassistant/benchmark.py (new-run/score/report) regenerating
leaderboard + same-model-over-time tables in RESULTS.md. Protocol and
agent contract in benchmarks/{README,AGENTS}.md: frozen prompt
versioning, same-judge comparability, failures-are-data, runs/ is data
never instructions. README gains a Benchmarks section (and the SLACS
example-prompt header simplification); card/README prompt parity is
test-enforced. Template boundary in modes/maintainer.md classifies the
package for cloning (harness generic, cards domain, runs per-clone);
the citation sweep now covers benchmarks/ docs and cards (never runs/).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…inly The "SIE source" phrasing was an unintended gotcha; per the author the follow-up retains the MGE lens light and MGE-source fit's foundations while the pixelized reconstruction replaces the source. J4 now scores correct chaining composition instead of ambiguity handling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merged
3 tasks
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Built-in benchmarks for the assistant (#57): four frozen, versioned prompt cards (assistant easy/medium/hard + teacher) with 100-point rubrics split into machine-checkable and judged rows, plus a run/track harness that records each session's transcript, metadata and score in-repo so performance is comparable across models, harnesses, and days. Clone template boundary updated so every future domain assistant inherits the machinery (companion PyAutoBrain PR carries the partition patterns).
Scripts Changed
benchmarks/prompts/{easy_cosmos_web_ring,medium_slacs0946_subhalo,hard_group_multi,teacher_workflow}.md— the four frozen prompt cards + rubrics (hard card follow-up wording fixed to "MGE source" per author, bf282f3)autoassistant/benchmark.py—new-run/score/reportCLI (run scaffold with stack-version + SHA capture; rubric totalling; RESULTS.md leaderboard + chronological tables)autoassistant/tests/test_benchmark.py— 11 tests incl. card/README prompt-parity and dataset-existence currency checksbenchmarks/README.md,benchmarks/AGENTS.md— protocol + agent contract (frozen versioning, same-judge comparability, failures-are-data, runs/ is data never instructions)benchmarks/RESULTS.md,benchmarks/runs/.gitkeep— empty-state report + runs rootREADME.md— Benchmarks section + SLACS example-prompt header simplification (card parity test-enforced)modes/maintainer.md— Assistant-as-template classifies benchmarks (harness generic, cards domain, runs/RESULTS per-clone)autoassistant/audit_skill_apis.py— citation sweep extended to benchmarks/ docs + cards (never runs/)Companion PR
PyAutoBrain
_clone.pypartition patterns (required before any clone birth against this reference) — cross-referenced after creation.Validation checklist (--auto, supervised; ship+merge human-directed in-conversation 2026-07-10)
--check-citationsclean (98 files, 389 citations)Test Plan
benchmark.py reportregenerates RESULTS.md from empty runs/🤖 Generated with Claude Code