feat(pinchbench): add PinchBench benchmark adapter - #244
Open
zeroasterisk wants to merge 3 commits into
Open
Conversation
Add benchmark adapter for PinchBench (pinchbench.com), a real-world benchmark for AI coding agents evaluating across 12 categories including productivity, coding, analysis, and memory. Signed-off-by: Alan Blount <alan@zeroasterisk.com>
Cover task-file parsing, manifest-driven task indexing, judge-JSON extraction, session lifecycle and automated/llm_judge/hybrid scoring, workspace staging, and evaluator aggregation. Tests build a small in-tree fixture skill repo, so they need no network, no clone of pinchbench/skill, and no LLM judge credentials. Signed-off-by: Alan Blount <alan@zeroasterisk.com>
zeroasterisk
force-pushed
the
feat/pinchbench-clean
branch
from
July 30, 2026 20:37
b805420 to
b0036d3
Compare
A live run against vertex_ai/gemini-2.5-flash always produced 0.0 with "LLM judge returned unparsable response". The hardcoded max_tokens=1024 was consumed almost entirely by reasoning tokens (observed: 979 reasoning / 41 output), so the verdict JSON was cut off mid-object and every task scored zero regardless of the answer. - raise the default judge budget to 4096, overridable via PINCHBENCH_JUDGE_MAX_TOKENS - report finish_reason == "length" as a distinct "truncated" note so a broken judge is not indistinguishable from a real 0.0 verdict - tolerate a None content payload instead of raising in the judge path Verified against the real judge afterwards: a strong answer scores 0.9 with a per-criterion breakdown, a weak one scores 0.0. Signed-off-by: Alan Blount <alan@zeroasterisk.com>
zeroasterisk
force-pushed
the
feat/pinchbench-clean
branch
from
July 30, 2026 21:09
299d6d0 to
40419fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add benchmark adapter for PinchBench, a real-world benchmark for AI coding agents. Evaluates across 12 categories: productivity, research, writing, coding, analysis, CSV analysis, log analysis, meeting analysis, memory, skills, and integrations.
Replaces #240, which accidentally bundled unrelated files. This PR is PinchBench-only.
Design
Follows the documented benchmark adapter pattern (
docs/adding-benchmarks.md):pinchbench_benchmark.py) with Session, Evaluator, and Benchmarkdatasets) imported lazily inside methods — host-importable without benchmark depsbash+submitFiles changed
src/exgentic/benchmarks/pinchbench/— benchmark adapter packagesrc/exgentic/interfaces/registry.py— registry entryValidation
ruff check+ruff formatcleanpy_compilepassesSigned-off-by: Alan Blount alan@zeroasterisk.com