Failsafe

Agent-first debugging CLI. Compresses noisy failure output into compact structured JSON packets so coding agents spend fewer tokens on debugging loops.

failure -> compact diagnosis -> minimal repro -> verify

Install

bun install

Requires Bun v1.0+. For installing Failsafe as a CLI, an MCP server, or a Claude Code skill, see docs/install.md. Release notes live in CHANGELOG.md.

Quick Start

# Initialize storage
failsafe init

# Run a command and capture the failure
failsafe run "pytest tests/"

# Get a structured diagnosis
failsafe diagnose last

# Create a minimal reproduction
failsafe repro last

# Verify after fixing
failsafe verify last

# Record the fix for the knowledge base
failsafe resolve last --success --fix-summary "Added null check"

All commands output JSON by default. Use --format text for human-readable output. Use --max-bytes to cap output size. On diagnose/explain, --evidence-only drops suggested fixes and next actions (keeping evidence, uncertainty, minimal_context, and a recomputed token_budget) for agents that prefer to reason for themselves.

Commands

Core

Command	Description
failsafe run <cmd>	Execute a command, capture output, return compact failure packet
failsafe diagnose <id|last>	Root-cause hypothesis with evidence and confidence
failsafe repro <id|last>	Extract a minimal reproduction (single test selector)
failsafe verify <id>	Re-run repro and original command to confirm fix
failsafe explain <id>	Combine all evidence into a compact explanation
failsafe init	Initialize .failsafe/ storage directory
failsafe config show|set	View or modify configuration
failsafe doctor	Check system dependencies
failsafe history	List past failures, find similar ones with --similar <id>

Debug (experimental)

failsafe debug <id> emits launch guidance — a ready-to-run command and breakpoint location for an interactive debugger you attach from your editor/IDE. It does not manage a live session. Python uses debugpy; Node.js uses the built-in V8 inspector (node --inspect-brk), which you attach to from VS Code ("Node: Attach") or chrome://inspect. Both pause execution and wait for the client to attach.

Command	Description
failsafe debug <id>	Emit a debugpy launch command + breakpoint for interactive debugging
failsafe step --session <id>	(experimental) In-process stepping; does not persist across invocations
failsafe inspect vars|stack|expr|source	(experimental) In-process inspection; does not persist across invocations

Debug sessions are in-memory within a single process, so step and inspect cannot reconnect from a separate CLI invocation — they return a structured debug_unavailable packet. For unsupported runtimes (Node.js, Go, Rust, Java, .NET), failsafe debug returns a structured packet naming the needed adapter and fallback commands (diagnose, repro).

Action budget and coarse-to-fine escalation

Every failsafe debug packet carries an action_budget: a plan that divides an action/token/time ceiling across the competing hypotheses and walks the tiers evidence → slice → breakpoint → step, cheapest first.

• Allocated by hypothesis. The diagnosed category is weighted by its own confidence and the residual belief becomes an explicit "something other than X" hypothesis, so a 0.4-confidence diagnosis does not quietly receive the whole budget. Every alternative gets at least one cheap look; no single hypothesis gets more than 60%.
• Gated on expected information gain. Line-level stepping is only authorized when a probe is expected to shift belief by at least 0.15 bits. Absent a caller-supplied probe model the plan assumes weak discrimination and stops at breakpoint, reporting stop_reason — escalation has to be argued for.
• Terminates by name. When a budget runs out the reason is one of actions_exhausted, tokens_exhausted, time_exhausted, information_gain_below_threshold, hypotheses_exhausted, or resolved, each with an actionable summary. "I ran out" is never confused with "I found it".

Override the ceiling with --budget <actions[,tokens[,ms]]> (default 24,12000,120000).

Hypotheses

failsafe hypotheses keeps the reasoning behind a localization instead of collapsing it to one category and a confidence number.

Command	Description
failsafe hypotheses build <id>	Build and persist the module → file → function → line tree plus a competing branch
failsafe hypotheses list <id>	Show the stored tree with its validation summary
failsafe hypotheses observe <id> <hyp-id> --outcome confirms|refutes|inconclusive --detail <text>	Record what a probe showed and update posteriors
failsafe hypotheses abandon <id> <hyp-id> --reason <text>	Drop a hypothesis, recording why

• Hierarchical. Refuting a file abandons every line hypothesis inside it, so an agent cannot keep probing a location it already ruled out.
• Falsifiable. Each hypothesis carries a probe and the observation expected under each outcome, written before the probe runs — a result cannot be reinterpreted afterwards to support whatever was already believed.
• Bayesian. Observations update posteriors through an explicit likelihood ratio (--likelihood-if-true / --likelihood-if-false) and renormalize across surviving siblings, so a refuted branch's belief moves to its competitors rather than evaporating.
• Intent-aware. Hypotheses record where their notion of correct behavior came from (spec, test, type, invariant, docstring, commit_message, inferred), and conflicting sources are surfaced rather than silently resolved.
• Explicitly abandoned. A reason is a required argument, not an optional field; the summary lists every dropped hypothesis with why it was dropped.

Design intent

failsafe intent <id> extracts what the code was supposed to do from every source that states it, and reports where those sources disagree.

Source	Read from
type	Python def f(x: int) -> Optional[str]: and TypeScript signatures, including parameter types
spec	Structured docstring/JSDoc tags: Returns:, Raises:, @returns, @throws
test	assert x == y, x is None, pytest.raises(E), expect(x).toBe(y), .toThrow(E), .toBeNull()
invariant	Runtime guards: assert x is not None, if not x: raise, if (!x) throw

Statements are normalized into comparable claims (returns, raises, nullable, param_type, equals) so a type promising Optional[str] and a docstring promising str register as a conflict. Extraction is conservative — only explicit syntactic forms, never free prose — because a fabricated contract is worse than a missing one.

Conflicts are surfaced, never resolved. The report includes an advisory source precedence, but nothing applies it: deciding whether the spec or the test is authoritative is a judgement about what the software is for. failsafe hypotheses build attaches the reconciled intent — primary source, location, and every contradicting statement — to the root hypothesis. --gate exits non-zero when sources conflict.

Tiered Rules

Command	Description
failsafe resolve <id>	Record fix outcome, update learned rules
failsafe rules list	List all rules (declared + learned + builtin)
failsafe rules show <id>	Show rule details and statistics
failsafe rules validate	Validate .failsafe/rules.yaml
failsafe rules export-learned	Export learned rules as YAML for promotion
failsafe rules disable <id>	Disable a learned rule
failsafe rules flaky	List flaky failure signatures
failsafe kb export	Export knowledge base to JSON
failsafe kb import <file>	Import knowledge base from JSON
failsafe kb export-dataset	Export resolved failure/fix pairs as JSONL training data

Tiered Rule System

Rules are evaluated in priority order:

1. Declared (.failsafe/rules.yaml) -- team-authored, project-specific
2. Learned (knowledge base) -- auto-generated from past resolutions
3. Built-in (12 templates) -- universal patterns shipped with Failsafe

Learned rules auto-promote when they reach sufficient confidence and occurrence count. Flaky tests are detected when failures recur after a fix.

MCP Interface

Failsafe ships an MCP (Model Context Protocol) server so flow orchestrators (AgentFlow, Statewright, etc.) can call it as a validation checkpoint. It exposes four tools over stdio, each returning the same JSON contract as the equivalent CLI command:

Tool	Equivalent CLI	Purpose
failsafe_analyze	run (+ diagnose if diagnose=true)	Run a command, capture/parse the failure, optionally diagnose
failsafe_diagnose	diagnose	Root-cause hypothesis for a stored failure
failsafe_repro	repro	Minimal reproduction selector
failsafe_verify	verify	Re-run repro + original to confirm a fix

Start the server:

failsafe-mcp        # installed binary
# or
bun run mcp         # from the repo

MCP client config example:

{
  "mcpServers": {
    "failsafe": { "command": "failsafe-mcp" }
  }
}

The CLI and MCP server share a single implementation (src/core/operations.ts), so their output contracts never diverge.

Telemetry (OpenTelemetry)

Telemetry is off by default. Set OTEL_EXPORTER_OTLP_ENDPOINT to emit OTLP/HTTP spans for the core operations:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces failsafe run "pytest tests/"

Spans emitted: failsafe.run, failsafe.parse, failsafe.diagnose, failsafe.repro, failsafe.verify. Attributes (prefixed failsafe.) include failure type, severity, root-cause category and confidence, parser matched, rule source, exit code, raw output bytes, and compression ratio. When the endpoint is unset there is zero overhead — the SDK is never loaded.

To line the spans up with agent-observability backends (Arize Phoenix, Langfuse), opt into the OpenTelemetry GenAI semantic conventions:

OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces failsafe diagnose last

Each span then also carries gen_ai.operation.name=execute_tool, gen_ai.tool.name (failsafe_analyze/_parse/_diagnose/_repro/_verify), gen_ai.tool.type=function, and — where a token budget exists — gen_ai.usage.input_tokens (what the raw output would have cost) and gen_ai.usage.output_tokens (what the compact packet costs). The failsafe.* set is unchanged; nothing gen_ai.* is emitted without the opt-in.

Content-capture policy

Every attribute is evaluated by a capture policy before it is written to a span, so the batch processor's buffer — and therefore every exporter — only ever holds cleared values. Three modes:

Mode	Emits
none	Span name, timing, and status only. No attributes.
metadata (default)	Allowlisted low-cardinality fields: counts, enums, confidences, schema version.
redacted-content	The above plus content values, each secret-redacted and truncated first.

Classification is deny-by-default: numeric and boolean values are metadata by construction, and a string value is content unless its key is on the canonical allowlist — so an attribute added anywhere in the codebase is withheld until it is explicitly classified.

Three ceilings bound the payload and the label cardinality a backend has to index: max_attribute_bytes per value (512), max_attributes_per_span (64), and max_attribute_cardinality distinct values per key (64). Anything withheld, truncated, redacted, or collapsed is counted, and the totals ride along on the span as failsafe.capture_dropped_fields, capture_truncated_fields, capture_redacted_fields, and capture_high_cardinality_fields — a dropped field stays observable as a number even when its value is not.

Configure in .failsafe/config.json under telemetry, or override the mode for a single run:

FAILSAFE_TELEMETRY_CAPTURE=none \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces failsafe run "pytest tests/"

Example .failsafe/rules.yaml:

version: "1"
rules:
  - id: "team-jwt-expired"
    pattern:
      error_contains: "token expired"
    diagnosis:
      category: "auth_error"
      explanation: "JWT expired. Auth service returns 422."
      fix: "Refresh token: POST /api/auth/refresh"
      enforcement: "suggest"
    confidence: 0.92

Output

Default output is JSON, optimized for agent consumption:

{
  "status": "failed",
  "failure_id": "fail_01HZX...",
  "summary": "KeyError: 'email' in create_user_from_oauth",
  "primary_location": { "file": "src/auth.py", "line": 42 },
  "test_summary": { "total": 18, "passed": 12, "failed": 6, "skipped": 0 },
  "raw_paths": {
    "stdout": ".failsafe/runs/fail_01HZX/stdout.log",
    "stderr": ".failsafe/runs/fail_01HZX/stderr.log"
  },
  "next": [
    { "command": "failsafe diagnose fail_01HZX", "reason": "Build a root-cause packet" }
  ],
  "token_budget": {
    "raw_output_bytes": 9231,
    "returned_bytes": 701,
    "compression_ratio": 13.2
  }
}

Output is capped by config.token_budget.max_output_bytes (default 6000). Use --max-bytes to override. When output is truncated, raw_paths point to the full untruncated files on disk. Use --format text for human-readable summaries.

Parsers

Built-in parsers (8 parsers across Python, JavaScript/TypeScript):

Language	Framework	What it extracts
Python	traceback	Stack frames, exception type, message
Python	pytest	Test names, assertion diffs, test summary, collection errors
JavaScript	stack trace	Stack frames, error type, application vs library frames
JavaScript	Jest	Test names, Expected/Received diffs, test summary
JavaScript	Vitest	Nested test paths, assertion diffs, test summary
TypeScript	tsc	TS error codes, file:line locations, total count
JavaScript	ESLint	Rule names, locations, problem count
JavaScript	Biome	Rule names, locations, error count

Plus Go, Rust, Java, Ruby, C/C++, and Mocha parsers. When a failing command's output matches no parser, a last-resort Drain-style template miner (drain-template) recovers the most failure-like log template, a file:line candidate, and a stable signature so unknown tools still produce a groupable, low-confidence diagnosis instead of "Unknown failure".

Trace ingestion (read-only)

src/trace/ turns a retrieved distributed trace into the same compact diagnosis packet as command output. Adapters normalize Jaeger JSON and OTLP/Tempo JSON into one span model (redacting attributes at ingest), root causes are ranked with the same causal graph used for multi-agent traces, and the packet carries trace_provenance. Every query must name a trace id — wildcard/unbounded scans are rejected — and both the lookback window and span count are capped. There is no write path to any backend.

Calibration

failsafe kb calibration --predictions <jsonl> checks whether a confidence number means anything: of the localizations called 0.8, roughly 80% should be right.

• Reliability curve — confidence binned against observed accuracy, with expected (ECE) and maximum (MCE) calibration error, a Brier score, and a signed bias that separates overconfident from underconfident.
• Top-k coverage — recall@1/3/5 and mean reciprocal rank, reported per granularity (module, file, function, line), because a system can be excellent at naming the file and useless at naming the line.
• Abstention — coverage plus the risk on the answered subset, and a selective gain figure. Declining to answer is only a virtue if what remains is more accurate; the report says whether it is.
• OOD slices — the same metrics cut by every tag in the data, so an out-of-distribution collapse nobody thought to look for still shows up.
• Recalibration — fitCalibration() fits a non-parametric histogram-binning map from raw confidence to observed accuracy. Bins with no data pass through unchanged rather than inventing a correction.

Abstentions are excluded from the reliability curve (a system that declined made no confidence claim), and a verdict is withheld below 30 answered predictions. --gate exits non-zero when confidences are overconfident.

Benchmarks

src/bench/ is pure: no dataset is downloaded, because fetching a corpus is a consequential external action. A user exports rows however they obtain them and the adapters map them into a canonical, pinned, versioned shape, so benchmark payloads never enter the repo or a release tar.

• Matrix (manifest.ts, runner.ts) normalizes SWE-bench-debug / SWE-smith / R2E-Gym rows into pinned instances, rejects unpinned commits and floating image tags, and appends results to JSONL so an interrupted sweep resumes instead of re-running.
• Service diagnosis (service-diagnosis.ts) scores diagnosing a running service across logs, traces, metrics, configuration, and source. Five dimensions are scored separately and never combined into one number: component localization (top-1, top-k, MRR, abstention), cause class (scored independently of localization, so "right cause, wrong service" is visible), explanation evidence (precision/recall/F1 with per-artifact recall and a count of citations to artifacts the case never supplied), latency, and cost. Availability slices report accuracy per evidence surface, which is where a system that only really reads logs becomes visible. A case with no prediction is scored as a full abstention rather than skipped.

Project memory (opt-in)

failsafe memory build indexes the repo's symbols, import graph, and test ownership into .failsafe/project-index.json; memory refresh re-hashes and rebuilds only what changed; memory status/memory query <id> inspect it. With memory.enabled: true in config, diagnose retrieves a byte-budgeted slice keyed by the failure's frames, symbols, and the files recent failed fixes touched, and records the ids/scores in a retrieval block. The index stores no file content — only paths, symbol names, imports, and hashes — and never indexes .env*, key/credential paths, or ignored directories.

Security

• Command policy: Commands are validated against an allowlist. Shell operators (&&, ||, ;, |) are split and each sub-command is checked. Shell metacharacters (backticks, $(...), ${...}) are blocked.
• Secret redaction: 16 patterns (OpenAI, Anthropic, GitHub, GitLab, Google, Slack, AWS, HF tokens, JWTs, PEM private keys, etc.) plus 30+ sensitive env var names are redacted before storage and output.
• Local-first: No cloud uploads, no telemetry, no external API calls.

Storage

Local-first under .failsafe/:

.failsafe/
  config.json          # Project configuration
  history.sqlite       # Failure records, diagnoses, learned rules, signatures
  runs/
    fail_01HZX/
      stdout.log       # Raw captured output
      stderr.log
      parsed.json      # Structured parse results
      diagnosis.json   # Diagnosis packet

Configuration

failsafe config show
failsafe config set token_budget.max_output_bytes 4000
failsafe config set security.allow_commands '["pytest","npm","bun"]'

Key settings:

Key	Default	Description
default_format	"json"	Output format
token_budget.max_output_bytes	6000	Max output size in bytes
security.allow_commands	16 common tools	Command allowlist
security.deny_patterns	rm -rf, sudo, etc.	Blocked command patterns
rules.auto_learn	true	Record failures for learning
rules.staleness_days	90	Days before a learned rule is flagged stale
timeouts.run_seconds	120	Command execution timeout

Agent Integration

Instruct your coding agent:

When a command fails, call Failsafe first:
1. failsafe run "<command>" to capture the failure
2. failsafe diagnose last before opening source files
3. failsafe repro last before stepping through code
4. failsafe verify last after applying a fix
5. failsafe resolve last --success after confirming the fix
Treat Failsafe output as the compact failure context. Only request raw logs when the diagnosis lacks evidence.

A Claude Code skill is included at skills/failsafe/ — copy to ~/.claude/skills/failsafe/ or .claude/skills/failsafe/ for automatic integration. See docs/install.md for details.

Development

bun install              # Install dependencies
bun test tests/          # Run tests (163 tests)
bun run test:e2e         # Run e2e tests against fixture projects
bun run typecheck        # TypeScript check
bun run lint             # Biome lint
bun src/cli/index.ts     # Run CLI directly

License

MIT