Summary
In the skill-eval exact-match verifier, the word-boundary regex can never match a ground_truth that ends (or starts) with a non-word character, so correct agent answers score 0.
Location
src/benchflow/templates/judge.py.tmpl, judge_exact_match():
if re.search(r'\b' + re.escape(ground_truth) + r'\b', content):
return 1.0Problem
\b (word boundary) requires a word character ([A-Za-z0-9_]) adjacent to the boundary position. When ground_truth ends in a non-word character — very common for code answers — the trailing \b cannot match. Examples that are impossible to score correctly:
pdfDoc.getPageCount() — ends in )pypdfium2 (Apache/BSD License) — ends in )mcp.run(transport="streamable_http") — ends in )
An agent that writes exactly the expected answer still scores 0.
Reproduce
import re
gt = "getPageCount()"
content = "The method is getPageCount()."
print(bool(re.search(r'\b' + re.escape(gt) + r'\b', content))) # False (expected True)
Suggested fix
Make the boundaries conditional on the ground_truth's own edge characters (only require \b where the adjacent gt character is a word char), or match on a looser delimiter set, e.g.:
left = r'(?<![A-Za-z0-9_])' if gt[:1].isalnum() or gt[:1] == '_' else ''
right = r'(?![A-Za-z0-9_])' if gt[-1:].isalnum() or gt[-1:] == '_' else ''
if re.search(left + re.escape(gt) + right, content):
return 1.0Found while evaluating skill packagings with bench skills eval; happy to send a PR if useful.
Summary
In the skill-eval exact-match verifier, the word-boundary regex can never match a
ground_truththat ends (or starts) with a non-word character, so correct agent answers score 0.Location
src/benchflow/templates/judge.py.tmpl,judge_exact_match():Problem
\b(word boundary) requires a word character ([A-Za-z0-9_]) adjacent to the boundary position. Whenground_truthends in a non-word character — very common for code answers — the trailing\bcannot match. Examples that are impossible to score correctly:pdfDoc.getPageCount()— ends in)pypdfium2 (Apache/BSD License)— ends in)mcp.run(transport="streamable_http")— ends in)An agent that writes exactly the expected answer still scores 0.
Reproduce
Suggested fix
Make the boundaries conditional on the ground_truth's own edge characters (only require
\bwhere the adjacent gt character is a word char), or match on a looser delimiter set, e.g.:Found while evaluating skill packagings with
bench skills eval; happy to send a PR if useful.