Targeted: 0.5.0 (reliability program). Baseline: release/0.3.0.
Four measurement gaps that keep real failures invisible:
- Non-zero exits count as success.
processes.py:280 hardcodes "ok": True for any command that ran; a failing test suite is a successful tool call and that is correct MCP semantics — the bug is that telemetry collapses both into one ok. Add an outcome dimension: exited_0 | exited_nonzero | timeout | signal | spawn_error, bucketed in tool_summary/session_end.
consecutive_failures is a single global slot (telemetry.py:400-407): any success of any tool resets it, so alternating failures never accumulate (17 consecutive apply_patch failures interleaved with 6 read_file successes peaked at 5 in the 0.2.x sample). Track per (tool, error_code), cleared only by a success of the same tool; bounded (LRU ~16).
- Summaries only flush at close. 0.3.0 made
tool_error immediate and documented that long-lived HTTP processes rarely emit session_end; add periodic incremental flush (every N calls or T minutes).
- One generic output schema for the whole catalog (
tool_output_schema()): command_id, next_action, truncated, status, exit_code are undeclared. Publish per-tool output schemas (also a prerequisite for the apply_changes program).
Privacy red lines unchanged from 0.3.0: closed schema, no paths/content, allow-listed clientInfo labels.
Targeted: 0.5.0 (reliability program). Baseline:
release/0.3.0.Four measurement gaps that keep real failures invisible:
processes.py:280hardcodes"ok": Truefor any command that ran; a failing test suite is a successful tool call and that is correct MCP semantics — the bug is that telemetry collapses both into oneok. Add anoutcomedimension:exited_0 | exited_nonzero | timeout | signal | spawn_error, bucketed intool_summary/session_end.consecutive_failuresis a single global slot (telemetry.py:400-407): any success of any tool resets it, so alternating failures never accumulate (17 consecutiveapply_patchfailures interleaved with 6read_filesuccesses peaked at 5 in the 0.2.x sample). Track per(tool, error_code), cleared only by a success of the same tool; bounded (LRU ~16).tool_errorimmediate and documented that long-lived HTTP processes rarely emitsession_end; add periodic incremental flush (every N calls or T minutes).tool_output_schema()):command_id,next_action,truncated,status,exit_codeare undeclared. Publish per-tool output schemas (also a prerequisite for theapply_changesprogram).Privacy red lines unchanged from 0.3.0: closed schema, no paths/content, allow-listed clientInfo labels.