Skip to content

feat(cli): add an opt-in failure-taxonomy report categorizing tool-failure causes for unattended runs (#222) - #223

Merged
qwen-code-dev-bot merged 1 commit into
mainfrom
issue/222-run-failure-taxonomy
Jul 22, 2026
Merged

feat(cli): add an opt-in failure-taxonomy report categorizing tool-failure causes for unattended runs (#222)#223
qwen-code-dev-bot merged 1 commit into
mainfrom
issue/222-run-failure-taxonomy

Conversation

@qwen-code-dev-bot

Copy link
Copy Markdown
Owner

Summary

Adds an opt-in (--failure-taxonomy) privacy-safe failure-taxonomy report for unattended (-p) runs, decomposed from roadmap #38 (observable autonomous execution).

The run summary (#40) reports a single terminal reason and counts tool failures by name, but not why each call failed — so operators triaging a failed run had to read raw logs to tell a policy denial from a hook denial, an approval denial, a path escape, or a tool error. This report categorizes each failed tool call into a fixed taxonomy and reports the terminal reason class.

Design

  • New src/run-failure-taxonomy.ts: a FailureTaxonomyCollector accumulates per-cause failure counts into a fixed taxonomy (policy_denied, hook_denied, approval_denied, folder_trust_denied, path_escape, unknown_tool, tool_error, plus an other bucket); build emits a bounded report in canonical order with the terminal reason. Mirrors the run-summary.ts/run-bottleneck.ts schema and redaction conventions.
  • src/agent.ts: executeToolCall records the cause at each failure branch (unknown tool, folder-trust, command policy, PreToolUse hook, approval), and categorizes a thrown error or returned isError result as path_escape (detected via a resolveSafe pre-check for mutate-file tools) or tool_error. Threaded through AgentOptions.
  • src/headless-protocol.ts: new failure_taxonomy NDJSON event.
  • src/index.ts: opt-in --failure-taxonomy flag wired into both the NDJSON and plain-text headless paths.

The report is metadata only — category names, counts, and the terminal reason. It has no channel for error text, prompts, file contents, secrets, or raw tool payloads.

Testing

  • tests/unit/run-failure-taxonomy.test.ts (10 tests): per-cause counting, canonical ordering, other bucket, non-zero-only output, reason sanitization, empty inventory, formatting.
  • tests/integration/run-failure-taxonomy.test.ts (5 tests): text/JSON emission, an unknown-tool failure categorized, a path-escape write categorized distinctly, secret-shaped arguments kept out of the report, zero-failure inventory.

Gates: typecheck, build, unit (1608), integration (+ desktop), smoke (51) all pass.

Closes #222

…ilure causes for unattended runs (#222)

The run summary reports a single terminal reason and counts tool failures by name, but not why each call failed, so operators triaging a failed run had to read raw logs to tell a policy denial from a hook denial, an approval denial, a path escape, or a tool error. Add a privacy-safe, bounded failure-taxonomy report (opt-in --failure-taxonomy) that categorizes each failed tool call into a fixed taxonomy and reports the terminal reason class, in both the NDJSON headless stream and the plain-text path. The report is metadata only (category names, counts, terminal reason) and has no channel for error text, prompts, file contents, secrets, or raw tool payloads.
@qwen-code-dev-bot

Copy link
Copy Markdown
Owner Author

E2E verification

  • Repository: qwen-code-dev-bot/oh-my-cli
  • Pull request: #223
  • Commit: 4e8910b2dd6440eb30e38ff487b6fd8385f1069f
  • Evidence mode: terminal
  • Result: PASS
Scenario Command Exit Duration
integration npm run test:integration 0 28.312s
smoke npm run smoke 0 3.016s

E2E evidence

Full immutable report

@qwen-code-dev-bot
qwen-code-dev-bot merged commit 638ff0e into main Jul 22, 2026
2 checks passed
@qwen-code-dev-bot
qwen-code-dev-bot deleted the issue/222-run-failure-taxonomy branch July 22, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run diagnostics: classify an unattended run's failures into a bounded, redacted failure taxonomy

1 participant