tools: schema trim + error-message audit (C4 + B3)#416
Merged
Conversation
Tool descriptions are fixed per-request overhead -- every byte rides on
EVERY provider call. The fattest ones were tutorials for formats the
error path already teaches:
* edit_file: the full hashline patch grammar (anchors, payload
markers, header hash) moves out of the description -- the format-
error messages (PreflightToolCall + the PasClaw.Hashline validators)
reply with the exact syntax the moment a patch is malformed, which
is the only time the model needs it.
* grep_files: the skip-list enumeration collapses to "skips VCS/build/
deps dirs and binaries"; gains the read_file start_line/end_line and
find_files cross-references (cheaper than the prose they replace and
they steer the next call).
* apply_patch: keeps the envelope shape, drops the tutorial; format
errors reply with the expected syntax.
* execute_code: keeps the callback contract, drops the fan-out story.
Measured (bench/agentloop, new build-site.first_request_bytes metric --
the fixed cost every single call pays):
before: 18720 after: 17774 (~240 tokens saved per request)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGQKz579j1ZnDRwmr6h1V6
Errors-as-documentation already worked here (the write_file truncation
error, the malformed-call nudge); this pass extends it to the errors a
coding agent actually loops on:
* no such file (read_file / edit_file): a bounded walk from the
working directory (3 hits, 400 dirs, error-path only) appends
"Did you mean: sub/deep.pas?" -- the dominant real case is right
name, wrong directory. Nothing similar -> points at find_files
with a concrete glob.
* unknown tool: TToolRegistry.RunTool appends up to 3 similar
registered names (containment or shared 4-char prefix; hidden
aliases excluded via Names) -- "unknown tool: read_files -- did
you mean: read_file?".
* apply_patch context-not-found: tells the model the context must
match the CURRENT file byte-exactly and to re-read the region with
read_file start_line/end_line before rebuilding the hunk.
* shell denylist refusals: name the rewrite ("without <token>") and
the dedicated-tool alternatives instead of just refusing.
Units: nearest-file suggestion (wrong dir), find_files fallback,
similar-tool suggestion, shell-denial remediation, apply_patch re-read
hint. Full bench + regression sweep green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGQKz579j1ZnDRwmr6h1V6
Drives a typo'd tool name (read_files) and a wrong-directory read
through the real dispatch path and asserts the errors carry the next
move ("did you mean: read_file", "Did you mean: pages/sub-dir-page.html").
On the pre-B3 binary both assertions fail (bare "unknown tool: X" /
"no such file: Y"), so the audit is now harness-verified, not just
unit-tested.
Fresh before/after against merged main (#415), same bench binary:
first_request_bytes: 18726 -> 17780 (C4, every request)
realtask total bytes: 296859 -> 286453 (trim x 11 requests)
error-guidance: 2 FAIL -> PASS (B3)
realtask deliverable: byte-identical index.html either way
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGQKz579j1ZnDRwmr6h1V6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last two items of the effectiveness queue, based on merged main (#415). Three commits: C4, B3, and the bench scenario that proves B3 end-to-end.
Harness-verified — same bench binary, before = merged main (#415), after = this branch
build-site.first_request_bytes(fixed cost every call pays)realtask.total_request_bytes(original pasclaw.dev prompt, 11 calls)error-guidancescenario (typo'd tool + wrong-dir read through real dispatch)unknown tool: X/no such file: Yrealtaskdeliverable (index.html)C4 — trim always-sent schemas; formats teach on the error path
edit_file: the full hashline patch grammar leaves the description —PreflightToolCall+ thePasClaw.Hashlinevalidators reply with the exact syntax the moment a patch is malformed, the only time the model needs it.grep_files: skip-list enumeration collapses; gainsread_file start_line/end_lineandfind_filescross-references.apply_patch: keeps the envelope shape, drops the tutorial.execute_code: keeps thepasclaw __toolcallback contract, drops the fan-out story.B3 — every tool error states the next move
no such file(read_file/edit_file) → bounded walk (3 hits, 400 dirs, error-path only) appends "Did you mean:pages/sub-dir-page.html?" — right name, wrong directory is the dominant real case. Nothing similar → a concretefind_filesglob.apply_patchcontext-not-found → context must match the current file byte-exactly; re-read the region withread_file start_line/end_linefirst.Tests
fs_tool_naming_testsgains the B3 section (nearest-file,find_filesfallback, similar-tool, shell-denial remediation,apply_patchre-read hint); newerror-guidancebench scenario fails on main and passes here. Full bench (7 scenarios) + regression sweep green on the rebased branch.🤖 Generated with Claude Code
https://claude.ai/code/session_01LGQKz579j1ZnDRwmr6h1V6