feat(cli): add an opt-in failure-taxonomy report categorizing tool-failure causes for unattended runs (#222) - #223
Merged
Conversation
…ilure causes for unattended runs (#222) The run summary reports a single terminal reason and counts tool failures by name, but not why each call failed, so operators triaging a failed run had to read raw logs to tell a policy denial from a hook denial, an approval denial, a path escape, or a tool error. Add a privacy-safe, bounded failure-taxonomy report (opt-in --failure-taxonomy) that categorizes each failed tool call into a fixed taxonomy and reports the terminal reason class, in both the NDJSON headless stream and the plain-text path. The report is metadata only (category names, counts, terminal reason) and has no channel for error text, prompts, file contents, secrets, or raw tool payloads.
Owner
Author
E2E verification
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in (
--failure-taxonomy) privacy-safe failure-taxonomy report for unattended (-p) runs, decomposed from roadmap #38 (observable autonomous execution).The run summary (#40) reports a single terminal reason and counts tool failures by name, but not why each call failed — so operators triaging a failed run had to read raw logs to tell a policy denial from a hook denial, an approval denial, a path escape, or a tool error. This report categorizes each failed tool call into a fixed taxonomy and reports the terminal reason class.
Design
src/run-failure-taxonomy.ts: aFailureTaxonomyCollectoraccumulates per-cause failure counts into a fixed taxonomy (policy_denied,hook_denied,approval_denied,folder_trust_denied,path_escape,unknown_tool,tool_error, plus anotherbucket);buildemits a bounded report in canonical order with the terminal reason. Mirrors therun-summary.ts/run-bottleneck.tsschema and redaction conventions.src/agent.ts:executeToolCallrecords the cause at each failure branch (unknown tool, folder-trust, command policy, PreToolUse hook, approval), and categorizes a thrown error or returnedisErrorresult aspath_escape(detected via aresolveSafepre-check for mutate-file tools) ortool_error. Threaded throughAgentOptions.src/headless-protocol.ts: newfailure_taxonomyNDJSON event.src/index.ts: opt-in--failure-taxonomyflag wired into both the NDJSON and plain-text headless paths.The report is metadata only — category names, counts, and the terminal reason. It has no channel for error text, prompts, file contents, secrets, or raw tool payloads.
Testing
tests/unit/run-failure-taxonomy.test.ts(10 tests): per-cause counting, canonical ordering,otherbucket, non-zero-only output, reason sanitization, empty inventory, formatting.tests/integration/run-failure-taxonomy.test.ts(5 tests): text/JSON emission, an unknown-tool failure categorized, a path-escape write categorized distinctly, secret-shaped arguments kept out of the report, zero-failure inventory.Gates: typecheck, build, unit (1608), integration (+ desktop), smoke (51) all pass.
Closes #222