feat(bench): nous-bench framework — Phase 1 + 2#291
Open
Naima2002 wants to merge 25 commits into
Open
Conversation
Phase 0 skeleton + Phase 1.1 contract for the nous-bench framework. Defines the Variant protocol and the VariantResult, Campaign, Experiment, and Budget dataclasses in bench/variants/base.py. No runtime behavior yet. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 1.2: first concrete campaign and experiment yamls plus tests covering Campaign.from_yaml and Experiment.from_yaml parsers and the optional max_wall_seconds field. 4/4 tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Loads experiment configs, validates requested variants, clones the target repo into an isolated workspace per variant, runs variants sequentially, and writes per-variant and combined result JSON files. Adds runner and isolation tests; 30/30 bench tests passing.
Adds markdown report generation from results.json and the python3 -m bench CLI with run/list commands. Wires report generation into the runner and adds tests for report rendering and CLI parsing.
Runs plain Claude Code headlessly with no methodology, parses JSON output into VariantResult, registers the variant in the runner, and adds parser tests for success and error cases.
Phase 2.2: bench/metrics.py centralizes parsing of `claude --print
--output-format json`. parse_claude_json normalizes to {final_answer,
tokens_in (billable only), tokens_out, dollars, is_error, subtype,
num_turns}. LLMMeter dataclass accumulates across multiple Claude calls
(used by Phase 4 loop variants). claude_plain refactors to use it.
Parser tests migrate to test_bench_metrics.py.
ThreadPoolExecutor swap (per agent plan-review note 1 — variants spawn their own subprocesses, threads avoid pickle without losing concurrency). Default --max-parallel-variants is min(N, cpu_count). Output order matches input order regardless of completion order. bench/judge.py + bench/judge_prompt.md: a separate Claude session scores each variant's final_answer on correctness (0-10) and completeness (0-10). Crashed variants are skipped (scores=None). Judge prompt is committed and pinned. Token cost in a top-level results.json:judge_usage field, never per-variant. Runner wires judge to run after variants complete; --skip-judge for debugging. New phase2_compare.yaml runs claude_plain + nous on blis_prefix.
Phase 2.7: smoke test revealed nous's final_answer was just discrepancy_analysis — a one-paragraph cross-arm summary stripped of all the per-arm experimental data (predicted/observed/status/diagnostic_note). The judge correctly scored that 3/2 because what it saw had no numbers, no mechanism, and no evidence — while claude_plain's full prose got 8/8. The bug was in extraction, not in the judge. _read_final_answer now renders the canonical nous structure (arms[] + discrepancy_analysis + experiment_valid) as a single comparison-friendly multi-section string when arms are present. The new output for the smoke run grows from 375 chars (summary only) to 2,393 chars including per-arm predictions, observations with real TTFT measurements, and status flags. 3 new tests cover the rich rendering + the fall-through path.
Phase 2 report integration: Correctness / Completeness columns added when judge scores are present in results.json, judge_usage shown in header, per-variant rationale section. Tokens column dropped
_read_final_answer now renders the full structured set nous produces: - All iter-N findings.json (not just latest), in ascending order - principles.json (cumulative cross-iteration knowledge) - ledger.json (per-iter outcome rows with prediction_accuracy) - report.md (when nous wrote one; gracefully skipped otherwise) Same shape as Phase 1.7.1 and Phase 2.7 fixes — closing the extraction-too-narrow loophole for good by widening once. Function signature changes to take artifacts_dir instead of runs_dir; call site in NousVariant.run() updated accordingly. 18 new tests covering each source + multi-iteration end-to-end + missing-source fall-through + double-digit iter ordering + lenient principle status filtering. 97/97 bench tests pass. Verified against runs/phase2_compare_with_fix/nous/.../ — all 3 RP-N principles + iter-1 findings + ledger row appear together (6,869 chars), and report.md absence is handled gracefully.
Collaborator
Author
added 7 commits
June 16, 2026 18:24
claude_plain.py shrinks from ~107 lines to ~25 by delegating subprocess + parsing + result aggregation to bench/variants/_claude_common.py. New helpers (ClaudeInvocation/ClaudeRunResult dataclasses, invoke_claude(), variant_result_from()) are designed to be reused by the three new variants in the next batches (claude_methodology, claude_loop, claude_methodology_loop). variant_result_from supports both single-call (claude_plain, claude_methodology) and multi-call loop variants. Aggregation rules: sum tokens/dollars/wall, last non-crashed final_answer, any crashed trips the variant, summed-total hit_cap, multi-run logs concatenated into one variant-level log file. DEFAULT_MODEL re-exported from claude_plain for backward compat. 18 new tests for the shared helpers covering each invoke_claude crash path and each variant_result_from aggregation rule. 115/115 bench tests pass (was 97; +18).
Single Claude Code session with methodology.md injected as --append-system-prompt. The L1 baseline: tests whether nous's methodology delivered as a prompt produces the same outcomes as nous's methodology delivered through orchestration. Implementation: thin wrapper over bench/variants/_claude_common.py's invoke_claude(), passing system_prompt=methodology_body. Reads bench/methodology/methodology.md on each call; fails fast with a clear error if the file is missing. Registered in runner.VARIANT_REGISTRY; `python3 -m bench list` now shows it alongside nous and claude_plain. 6 new tests: variant identity, methodology threading into system_prompt, graceful crash on missing methodology.md, metric pass-through, crash propagation, and a sanity check that METHODOLOGY_PATH points at the committed file. 121/121 bench tests pass.
ClaudeLoopVariant runs claude_plain N times sequentially (N from
budget.max_iterations). Each call after the first prepends the previous
session's full final_answer to the user message with a clear separator;
no methodology, no principle extraction, no schema enforcement. Tests
'iteration alone' — does running more times help, or does the agent drift
without structure?
Loop short-circuits on first crash. Per-iter logs land at
workspace/.bench-claude-loop-iter-{n}.log; variant_result_from
concatenates them into a single workspace/.bench-claude_loop.log at the
end. Aggregation per the shared rules in _claude_common: sum
tokens/dollars/wall, last non-crashed answer.
Registered in runner.VARIANT_REGISTRY. `python3 -m bench list` now
shows 4 variants.
11 new tests covering: variant identity, max_iterations call count,
short-circuit on crash, first-iter question pass-through, prepend logic
on subsequent iters, no-history-accumulation (only most recent), metric
aggregation, single-iter degenerate case, distinct per-iter log paths,
last-non-crashed final_answer, partial-crash recovery. 132/132 bench
tests pass.
Final batch closing #293: - bench/experiments/phase4_full_sweep.yaml: runs all 5 variants on blis_prefix in parallel (live smoke costs ~$15-25; not run as part of this PR) - docs/bench/walkthrough.md: comprehensive overview — what the bench does, three-layer model, what the user sees, what happens in the background, all 5 variants explained, parallel execution, output layout, token accounting, the 1.7.1 / 2.7 / 2.8 pattern, test architecture (161 tests), design choices - docs/bench/variants.md: variant Protocol + VariantResult contract, the 5 currently-registered variants and what each isolates, 5-step recipe for adding a new variant, standards every variant must follow (token accounting, final_answer fidelity, no live LLM calls in tests, behavioral testing only) - README.md: new 'Benchmark harness' section before 'Project Structure' pointing at docs/bench/; Project Structure tree updated to include bench/ subdirectories - docs/architecture.md: brief 'Bench harness' subsection at end pointing at docs/bench/walkthrough.md 161/161 bench tests still pass. Sub-issue #293 acceptance fully met: - All 5 variants registered, `python3 -m bench list` shows them - phase4_full_sweep.yaml in place - methodology.md committed and untouched - User-facing docs added under docs/bench/
… w/ principle carry-forward (#293) Should have landed before commit 653a15b (the docs commit). The variant file, its test file, and the runner.py registry registration were left uncommitted, leaving PR #291 in an inconsistent state where docs and phase4_full_sweep.yaml reference claude_methodology_loop but the runner doesn't register it. Fresh clones would crash with 'Unknown variants: [claude_methodology_loop]'. PR review (workflow Step 7) caught this. ClaudeMethodologyLoopVariant runs claude_methodology N times sequentially with deterministic principle extraction (regex-based) and carry-forward between sessions. Principles accumulate across iterations without nous's structural enforcement (no schema validation, no dedup, no conflict detection) — that absence is the L2 experimental signal. Includes _extract_principles regex with 17 edge-case tests + variant wiring tests. 161/161 bench tests pass.
Collaborator
Author
|
Sub-issue B (#293) closed — bench framework has all 5 variants live-validated on iter=1 ladder: L0 8/7 → L0+iter 7/8 → L1 9/9 → L2(degenerate) 9/9 → L3 8/8 ( Headline: at iter=3, L3's complexity-tier discipline forced hypothesis-family escalation (prefix-cache-ttft → dose-response → kv-pressure), surfacing a 60× TTFT regime change at constrained KV cache. The prompt-based variant did not find this — it parameterized the original hypothesis along input length instead. Judge's current 2-axis rubric (correctness + completeness) understates this asymmetry. Next: 11-metric selectable judge rubric (separate sub-issue), then 5-system L0/L1/L2/L3 ablation for the paper. |
added 5 commits
June 18, 2026 12:02
#295) judge_prompt.md is now a frame template; per-metric rubric language lives in METRIC_RUBRICS in judge.py. 11 metrics: correctness, completeness, novelty, coverage, diagnostic_value, reproducibility, iter_coherence, principle_yield, causal_explanation_depth, transferability, structured_artifact_production. 6 named presets: default, ablation-single-iter, ablation-multi-iter, case-study, transferability, minimal. Selectable via --judge-metrics (csv) and/or --judge-preset. iter_coherence auto-dropped on single-iter runs. New 'bench rejudge' subcommand re-runs the judge over an existing results.json without re-running variants — lets us iterate on the rubric cheaply. New 'bench import-nous' subcommand converts an existing nous campaign directory into a bench-compatible results.json. Auto-detects flat vs nested artifact layouts. Path-agnostic so anyone helping with experiments can run it on their machine. Optional --merge-baselines flag combines imported nous result with an existing baselines run for unified rejudging. JudgeScore.scores is now dict[str, int|None] for variable metrics; convenience accessors .correctness and .completeness preserved for backward compat. Validation: rejudged phase4_iter3 under all 11 metrics across Sonnet (3x) and Opus (1x). Rubric directionally captures structural-enforcement wins for nous (reproducibility +3, structured_artifact_production +3, iter_coherence +2, causal_explanation_depth +2) vs methodology's coverage win (-2 completeness, -1 coverage). Inter-rater spread: 8/11 metrics within +/-1 across 3 Sonnet runs; 3 metrics drift to +/-2 (completeness, coverage, structured_artifact). Also fixes a pre-existing bug in _harvest_metrics where 'cost_usd: null' rows in some campaigns crashed the parser. bench/experiments.md added — the Component 1 ablation plan: 5 systems, how to run each, where to look after. Tests: 240 passing (was 161). New test files: test_bench_import_nous.py (20 tests). Test seam unchanged: no live LLM calls.
Replaces strict-format principle extraction with permissive takeaway extraction. Rationale: the structured principle store with regime / mechanism / applicability_bounds / confidence is nous's novelty; giving the L2 baseline a regex-knockoff version of it understates nous's structural-enforcement claim in the L0/L1/L2/Nous ablation. Changes: - methodology.md: soft suggestion to end with 'Key takeaways from this iteration' bullet section. Replaces the strict ## Principles requirement. - _extract_principles renamed to _extract_takeaways (alias kept for backward compat). Heading regex broadened to match: Key Takeaways, Takeaways, What I Learned, Lessons Learned, Key Findings, Summary, plus Principles for back-compat with the old methodology. - New fallback: when extraction returns empty, prepend the FULL previous iter answer to the next iter's prompt (same mechanism as claude_loop). Guarantees claude_methodology_loop >= claude_loop on memory richness. - Extracted _build_next_question helper for testable prompt construction. Tests: +15 (45 total in this file). Pin: each new heading variant, back-compat with Principles heading, unrelated headings (Conclusion) don't match, [T1] ID prefix stripping works same as old [P1], full-answer fallback path triggers on empty extraction, takeaway-present path does NOT dump full answer.
…ut L2 (#295) Pivot the L0/L1/L2 prompt design to match what an engineer would actually paste into Claude. Methodology is inlined in the user prompt instead of passed via --append-system-prompt; L2's carry-forward is the full text of all prior iterations pasted under a '--- Prior findings ---' marker instead of regex-extracted takeaways. L0 (claude_plain) prompt: <research_question> Investigate this and report your findings. L1 (claude_methodology) prompt: <research_question> Approach this systematically: - Form hypotheses before running experiments. Predict what you expect to observe and why. - Run controlled experiments. - When your prediction is wrong, think carefully about why. - Track what you learn. Report your findings with the evidence that supports them. L2 (claude_methodology_loop) iter-1 prompt: <research_question> Approach this systematically: - Form hypotheses before running experiments. ... - Run controlled experiments. - When your prediction is wrong, ... - Track what you learn. Build on prior findings rather than starting from scratch each time. This is iteration 1 of N. Report your findings and what you would investigate next. L2 iter-K (K>1) prompt: same as iter-1, then 'This is iteration K of N. Here is what you found in prior iterations: --- Prior findings --- ... --- Continue your investigation. Build on what you found before.' The 'Build on prior findings rather than starting from scratch each time' clause is L2-only and lives in a separate methodology_loop.md, since L1 is single-session and the cross-iteration framing would be off-message. Changes: - methodology.md: stripped to four short scientific bullets (L1 form). - methodology_loop.md: NEW. Same four bullets but the last carries the cross-iteration 'Build on prior findings' clause. - claude_plain.py: appends 'Investigate this and report your findings.' to the research question for L0. - claude_methodology.py: drops --append-system-prompt; builds L1 user prompt with methodology inlined and the 'Report your findings' closing. - claude_methodology_loop.py: drops _extract_takeaways and the fallback branching entirely; per-iter builders inline methodology_loop.md, add iteration framing, and paste all prior iters' full output verbatim. Reads from METHODOLOGY_LOOP_PATH (the new file), not METHODOLOGY_PATH. - experiments.md: scrubbed of machine-specific paths. Now uses BENCH_NOUS_CAMPAIGNS / BENCH_BLIS_REPO / BENCH_GRAPH_COLORING_REPO / BENCH_FLINK_REPO env vars so anyone can follow the doc verbatim. Tests: 229 passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nousvariant wrappingnous runsubprocess (Phase 1)claude_plainvariant wrapping headless Claude Code (Phase 2)ThreadPoolExecutorparallel runner with--max-parallel-variantsflagbench/judge_prompt.mdblis_prefixvalidated end-to-end (~$11 spend); both surfaced real comparison-fairness bugs that unit tests couldn't catchRelated Issue
Tracks #283. Living draft PR for benchmark tracking —
bench-frameworkkeeps accumulating commits as sub-issues land:principles.jsonin nousfinal_answerfor fair judgingclaude_methodology,claude_loop,claude_methodology_loop(+ bootstrappedmethodology.md)Test Plan
python3 -m pytest tests/test_bench_*.py→ 79 passedpython3 -m bench listshows registered campaigns + variantspython3 -m bench run bench/experiments/phase2_compare.yamlsmoke run onblis_prefixwith claude_plain + nous in parallel + judge/pr-review-toolkit:review-prround once sub-issues close🤖 Generated with Claude Code