fix(skill-eval): match punctuated exact answers

[bingran-you]

Summary

Fixes #897 by changing skill-eval's exact-match fallback from fixed \b...\b word-boundary matching to word-character lookaround delimiters. This keeps exact answers standalone without making code-like punctuation such as trailing ) impossible to match.

Context

The generated skill-eval judge template currently uses a fixed word-boundary regex around ground_truth. That cannot match valid ground truths ending in non-word characters, including examples like getPageCount() or mcp.run(transport="streamable_http"). A correct agent answer can therefore score 0.0 in no-rubric exact-match mode.

Validation

• uv run python -m pytest tests/test_skill_eval_sweep.py::TestExactMatchJudgeDelimiters
• uv run python -m pytest tests/test_skill_eval_sweep.py
• uv run ruff check src/benchflow/templates/judge.py.tmpl tests/test_skill_eval_sweep.py

Notes

A follow-up commit will update the regression docstring with this PR number after GitHub assigns it.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[bingran-you]

Users Simulation automation review (2026-07-07): blocked on current head 95de7752c938b21d57ef9c06214c5f33b12d35be.

The intended skill-eval fix itself validated, but the branch is not merge-ready because a Daytona DinD regression is present on the same head.

Evidence:

• Focused skill-eval checks passed for the exact-match path. A local SDK proof generated a one-case skill/judge and scored The method is getPageCount(). as 1.0 while rejecting getPageCount()Wrapper as 0.0.
• Existing Daytona DinD coverage now fails: tests/test_daytona_dind_compose_up.py::test_long_exec_heartbeat_returns_promptly_after_fast_command times out under the new background sleep/trap heartbeat logic. This regresses the fast-command path the test guards.
• Live model canary did not produce publishable trajectory evidence: /tmp/benchflow-pr900-smoke3/2026-07-07__05-13-16 had provider errors, zero tokens/tool use, and no usable assistant completion. Oracle fallback at /tmp/benchflow-pr900-oracle/2026-07-07__05-17-45 passed the benchmark but is non-model evidence and correctly fails the hard publishability gate because oracle lanes do not emit trajectory/llm_trajectory.jsonl.
• The current GitHub integration-light-jobs artifact for this head does validate as 1/1 healthy (jax-computing-basics, OpenHands, glm/glm-5.1, ACP+LLM trajectories, results.jsonl training-ready, reward 1.0, 365,467 tokens), but its GLM user-endpoint cost_usd / price_source are null. That leaves the strict USD-cost metadata requirement only partially covered.

Moving to status:blocked; please fix the Daytona heartbeat regression before this can return to ready.

[bingran-you]

Daily scan follow-up (2026-07-08): resolved the stale Daytona DinD blocker by non-force updating the branch with current origin/main.

New head: 4f5f5b2e178ab55ce49e392a33cbb04e86a3b57e.

Validation in isolated worktree /tmp/benchflow-daily-pr900:

• uv sync --extra dev --locked
• uv run python -m pytest tests/test_skill_eval_sweep.py tests/test_daytona_dind_compose_up.py::test_long_exec_heartbeat_returns_promptly_after_fast_command tests/test_process.py -q -> 52 passed
• uv run ruff check src/benchflow/templates/judge.py.tmpl tests/test_skill_eval_sweep.py src/benchflow/sandbox/process.py tests/test_process.py -> passed

Moved labels back to status:ready + review:pending. This still needs refreshed GitHub CI and non-author review before merge.

[bingran-you]

Users Simulation automation review (2026-07-08): ready by simulation on head 4f5f5b2e178ab55ce49e392a33cbb04e86a3b57e.

PR-specific user simulation validated the exact-answer judge boundary:

• The method is getPageCount(). now scores 1.0.
• The method is getPageCount()Wrapper. stays 0.0.
• The same punctuated positive case scores 0.0 with the origin/main judge template, proving the PR fixes the regression.

Checks/evidence:

• Focused skill-eval checks passed: tests/test_skill_eval.py, tests/test_skill_eval_sweep.py, tests/test_skill_eval_dryrun.py.
• Static/CLI gates passed in the PR simulation: ruff, ty, bench --help, bench skills eval --help.
• Shared base sweep passed: ruff check ., ty check src, and 149 CLI/SDK/task/usage/skill tests.
• Current-head GitHub integration-light-jobs artifact validated with .agents/skills/benchflow-experiment-review/scripts/validate_run_artifacts.py: ACP trajectory, LLM trajectory, and training-ready results.jsonl are present; reward 1.0, 406,349 tokens, 21 tool calls, total timing 393.2s.

Thermo-nuclear review: no structural blocker found; this is a small template/test diff and does not add broad branching or file-size debt.

Metadata caveat: the GLM user-endpoint artifact has healthy token/timing metadata, but result.json.agent_result.cost_usd / price_source are still null. That is not specific to this PR, but it means GLM artifacts still do not fully satisfy the stricter non-null USD-cost production-data requirement.

Labels: status:ready / review:pending.

[bingran-you]

Users Simulation automation review (2026-07-10): ready by simulation on head 4f5f5b2e178ab55ce49e392a33cbb04e86a3b57e.

Verified:

• pytest tests/test_skill_eval_sweep.py -k ExactMatchJudgeDelimiters -q
• full tests/test_skill_eval_sweep.py
• benchflow --help, benchflow eval --help, benchflow skills --help, benchflow tasks --help
• benchflow tasks check src/benchflow/demo_task
• benchflow tasks check tests/examples/hello-world-task
• SDK smoke with TaskPaths(...)
• raw Gemini provider preflight

The requested Codex/OpenAI route was blocked in this workspace by unusable credentials/auth, and DeepSeek reported insufficient balance, so I used the cheapest working fallback path I could validate: openhands + gemini-3.1-flash-lite-preview on Docker.

Healthy artifact produced:

• Jobs root: /tmp/benchflow-pr900-openhands-gemini-jobs
• Rollout: hello-world-task__f5c5b925
• trajectory/acp_trajectory.jsonl: present
• trajectory/llm_trajectory.jsonl: present
• results.jsonl: one training-ready row
• result.json: token/cost/timing metadata present (total_tokens=25209, cost_usd=0.00079715, timing.total=79.7s)
• benchflow-experiment-review validator passed

Thermo-nuclear review: no blocking structure issue. The judge template change is narrow and direct; the regression test is focused and below the file-size smell threshold.

[bingran-you]

Users Simulation automation review (2026-07-11): ready by simulation on head 4f5f5b2e178ab55ce49e392a33cbb04e86a3b57e.

Scope checked: skill-eval exact-match punctuation boundary and broad CLI/SDK regression risk.

Evidence:

• Focused local gate passed after standard setup: uv sync --extra dev --locked; pytest tests/test_skill_eval_sweep.py -q -> 26 passed; focused ruff; ty check src; bench --help; SDK Scene.single(agent="oracle").
• Subagent extra exact-match smoke passed: standalone getPageCount(). is accepted, word-prefix / word-suffix wrappers are rejected, and normal word-boundary cases still behave.
• Additional CLI smokes from subagent passed: bench agent list, bench tasks check --help, bench eval run --help, bench skills eval --help, and bench tasks check tests/examples/hello-world-task.
• GitHub checks for this current head are green.

Artifact health:

• Current-head integration-light run 28929203493 validated healthy with benchflow-experiment-review: 1/1 healthy, ACP + LLM trajectories present, training-ready results.jsonl, reward present, 406,349 tokens, 21 tool calls, 393.2s timing.
• Caveat: this GLM user-endpoint artifact still has null agent_result.cost_usd / price_source; token, timing, and trajectory metadata are complete.

Thermo review: no structural blocker. The PR is a small direct judge-template change with targeted regression coverage; no file crosses 1k lines, no ad-hoc branch growth, no wrong-layer logic, and no thin wrapper concern.