Skip to content

Telemetry: distinct command outcomes, per-kind failure streaks, periodic flush, per-tool output schemas #63

Description

@cf-pages

Targeted: 0.5.0 (reliability program). Baseline: release/0.3.0.

Four measurement gaps that keep real failures invisible:

  1. Non-zero exits count as success. processes.py:280 hardcodes "ok": True for any command that ran; a failing test suite is a successful tool call and that is correct MCP semantics — the bug is that telemetry collapses both into one ok. Add an outcome dimension: exited_0 | exited_nonzero | timeout | signal | spawn_error, bucketed in tool_summary/session_end.
  2. consecutive_failures is a single global slot (telemetry.py:400-407): any success of any tool resets it, so alternating failures never accumulate (17 consecutive apply_patch failures interleaved with 6 read_file successes peaked at 5 in the 0.2.x sample). Track per (tool, error_code), cleared only by a success of the same tool; bounded (LRU ~16).
  3. Summaries only flush at close. 0.3.0 made tool_error immediate and documented that long-lived HTTP processes rarely emit session_end; add periodic incremental flush (every N calls or T minutes).
  4. One generic output schema for the whole catalog (tool_output_schema()): command_id, next_action, truncated, status, exit_code are undeclared. Publish per-tool output schemas (also a prerequisite for the apply_changes program).

Privacy red lines unchanged from 0.3.0: closed schema, no paths/content, allow-listed clientInfo labels.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions