Overview
First real runs of the shipped benchmark suite (#57, PRs #58/#79): execute the two cheap benchmarks against multiple models, record every run per the protocol, regenerate RESULTS.md, and deliver a calibration verdict on the rubrics themselves. Human-directed launch 2026-07-10 (in-conversation, continuing the #57 session); effective autonomy supervised (research cap).
Plan
- Branch
feature/benchmark-calibration in-place; all run records committed there and PR'd at the end.
- Run teacher-basic-workflow and assistant-easy-cosmos-web-ring against 2+ models on the claude-code Agent-tool harness (recorded honestly as
claude-code-subagent): planned matrix — teacher × {sonnet, haiku}, easy × {sonnet}, extending if budget/memory allow. Sessions run serially (fits need ~5.5 GB; laptop shared).
- Each benchmarked session is a fresh subagent operating inside the assistant clone under its normal instructions, with the card prompt pasted verbatim as the first user message; the operator (this session) replies honestly and minimally via SendMessage — no coaching, no rubric leakage. Failures are recorded, not discarded.
- Scaffold via
benchmark.py new-run, save the full operator-visible conversation as transcript.md, fill meta.yaml (hardware, duration, judge = claude-fable-5 for judged rows), score with evidence, benchmark.py report.
- Deliverables: committed run records + regenerated
RESULTS.md + a rubric-calibration comment on this issue (unscoreable/ambiguous/gameable rows → version-bump proposals, never in-place card edits).
Known harness caveat (recorded in each meta.yaml)
Subagent sessions are driven through the Agent tool rather than an interactive terminal; pacing-related judged rows (teacher J3) are scored against how the agent handles the turn-based exchange this allows.
Original Prompt
Filed at PyAutoMind/research/autolens_assistant/benchmark_calibration_runs.md (see repo history for verbatim text; scope = at least the two cheap benchmarks on 2+ model×harness combos, medium/hard deferred with a runtime note).
Overview
First real runs of the shipped benchmark suite (#57, PRs #58/#79): execute the two cheap benchmarks against multiple models, record every run per the protocol, regenerate
RESULTS.md, and deliver a calibration verdict on the rubrics themselves. Human-directed launch 2026-07-10 (in-conversation, continuing the #57 session); effective autonomy supervised (research cap).Plan
feature/benchmark-calibrationin-place; all run records committed there and PR'd at the end.claude-code-subagent): planned matrix — teacher × {sonnet, haiku}, easy × {sonnet}, extending if budget/memory allow. Sessions run serially (fits need ~5.5 GB; laptop shared).benchmark.py new-run, save the full operator-visible conversation astranscript.md, fillmeta.yaml(hardware, duration, judge = claude-fable-5 for judged rows), score with evidence,benchmark.py report.RESULTS.md+ a rubric-calibration comment on this issue (unscoreable/ambiguous/gameable rows → version-bump proposals, never in-place card edits).Known harness caveat (recorded in each meta.yaml)
Subagent sessions are driven through the Agent tool rather than an interactive terminal; pacing-related judged rows (teacher J3) are scored against how the agent handles the turn-based exchange this allows.
Original Prompt
Filed at
PyAutoMind/research/autolens_assistant/benchmark_calibration_runs.md(see repo history for verbatim text; scope = at least the two cheap benchmarks on 2+ model×harness combos, medium/hard deferred with a runtime note).