Targeted: 0.5.0 (reliability program). Baseline: release/0.3.0.
Four measurement defects that keep real failures invisible:
- Non-zero exits count as success.
processes.py:280 hardcodes "ok": True for any command that ran; test/build failures never reach the dashboards. Keep ok = "the tool call worked", add an outcome dimension (exited_0 | exited_nonzero | timeout | signal | spawn_error) and bucket tool_summary/session_end by it.
_failure_streak is one global slot cleared by any success (telemetry.py:400-407): 17 consecutive apply_patch failures interleaved with successful read_file calls never exceeds a streak of 5. Track per (tool, error_code) (bounded, LRU ~16), cleared only by a success of the same tool.- Summaries only flush at
finish(). A long-lived HTTP process emits session_end once at shutdown (documented in 0.3.0); add periodic/incremental flush (every N calls or T minutes) so aggregate data survives.
- One shared output schema for the whole catalog (
tool_output_schema()): command_id, next_action, truncated, status, exit_code are undeclared. Give each tool a real output schema — also a prerequisite for the structured-edit tool.
Privacy red lines carried over from 0.3.0: closed schema, no paths/content, sanitized clientInfo only.
Targeted: 0.5.0 (reliability program). Baseline:
release/0.3.0.Four measurement defects that keep real failures invisible:
processes.py:280hardcodes"ok": Truefor any command that ran; test/build failures never reach the dashboards. Keepok= "the tool call worked", add anoutcomedimension (exited_0|exited_nonzero|timeout|signal|spawn_error) and buckettool_summary/session_endby it._failure_streakis one global slot cleared by any success (telemetry.py:400-407): 17 consecutiveapply_patchfailures interleaved with successfulread_filecalls never exceeds a streak of 5. Track per(tool, error_code)(bounded, LRU ~16), cleared only by a success of the same tool.finish(). A long-lived HTTP process emitssession_endonce at shutdown (documented in 0.3.0); add periodic/incremental flush (every N calls or T minutes) so aggregate data survives.tool_output_schema()):command_id,next_action,truncated,status,exit_codeare undeclared. Give each tool a real output schema — also a prerequisite for the structured-edit tool.Privacy red lines carried over from 0.3.0: closed schema, no paths/content, sanitized clientInfo only.