Agent reliability layer: circuit breakers, loop detection, and automatic model escalation

[weklund-agent]

Summary

Implement an agent reliability layer directly in the serving infrastructure that detects common failure modes (infinite loops, malformed tool calls, runaway token usage) and automatically recovers — either by retrying with corrective prompts, escalating to a more capable model tier, or falling back to cloud. This makes any local model dramatically more reliable for autonomous agent workloads without requiring changes to the agent framework.

Problem

The #1 reason people give up on local models for autonomous agents is unreliable tool calling and instruction following. This is well-documented across communities:

• CrewAI forums: "agents frequently attempted tool calls using placeholder data," "agents produce garbage output or incorrect JSON formatting"
• Local model users consistently report: agent generates malformed JSON → retries → malformed again → infinite loop burning tokens and compute all night
• Research shows 73% of enterprise agent deployments hit reliability failures in their first year. On local hardware, failure modes are worse because smaller models have weaker instruction following

Nobody is solving this at the serving layer. Every agent framework expects the model API to "just work" and handles errors poorly. The agent framework generates a prompt, sends it to the API, and trusts the response. When the response is garbage, the framework either crashes, retries blindly, or loops forever.

Failure Modes in the Wild

1. Infinite tool-call loops: Model generates {"function": "search", "arguments": {"query": "..."}} → gets result → generates the exact same tool call → gets the same result → loops indefinitely. The agent framework doesn't detect this because each response is technically valid.

2. Malformed function calls: Model generates almost-valid JSON but with wrong field names, missing required fields, or extra trailing text. The agent framework's parser fails, it retries, the model generates the same malformed output.

3. Hallucinated tool names: Model calls execute_code when the available tools are run_terminal_command and write_file. Agent framework returns "unknown tool" error, model tries again with the same hallucinated name.

4. Runaway token consumption: Agent enters a reasoning loop, generating thousands of tokens of "thinking" that goes nowhere. On a cloud API this costs money; on local hardware it monopolizes the model server for hours.

5. Quality degradation under load: When memory pressure causes swap, model quality drops silently. The model starts generating incoherent responses, but the health check still passes because the server is technically running.

Proposed Solution

Architecture

The reliability layer sits between LiteLLM and the client (agent framework), implemented as a LiteLLM middleware or a thin proxy layer:

Agent Framework
       ↓
┌─────────────────────────┐
│   Reliability Layer     │
│  ┌───────────────────┐  │
│  │  Loop Detector     │  │
│  │  Tool Validator    │  │
│  │  Token Budget      │  │
│  │  Escalation Engine │  │
│  └───────────────────┘  │
└─────────────────────────┘
       ↓
   LiteLLM Proxy (:4000)
       ↓
   Model Tiers

Component 1: Loop Detection

Mechanism: Maintain a rolling window of the last N responses per session (identified by a session header or API key). Compute a similarity hash of each response (normalized, whitespace-stripped). If 3+ responses in the window produce the same or near-identical output, trigger circuit breaker.

Implementation:

class LoopDetector:
    def __init__(self, window_size=10, threshold=3, similarity=0.95):
        self.sessions = {}  # session_id -> deque of response hashes

    def check(self, session_id: str, response: str) -> LoopStatus:
        # Hash the normalized response
        # Check against rolling window
        # Return NORMAL, WARNING, or CIRCUIT_BREAK

On circuit break:

1. Return a structured error response that signals the agent framework to try a different approach:

   {
     "error": {
       "type": "loop_detected",
       "message": "Repeated identical output detected (3x). Consider rephrasing the request or trying a different approach.",
       "retry_after": 5
     }
   }

2. Optionally: auto-escalate to the next tier (see Component 4) instead of returning an error.

Configuration:

reliability:
  loop_detection:
    enabled: true
    window_size: 10
    duplicate_threshold: 3
    similarity_threshold: 0.95   # fuzzy matching for near-identical responses

Component 2: Tool Call Validation

Mechanism: When the request includes tools or functions definitions (OpenAI function calling format), validate the model's response against the provided schemas before returning it to the client.

Validation checks:

1. JSON validity: Is the tool call valid JSON?
2. Schema compliance: Do the arguments match the declared parameter types and required fields?
3. Tool existence: Does the called function name exist in the provided tool list?
4. Argument plausibility: Are required arguments present and non-empty?

On validation failure:

1. Inject a corrective system message and retry on the same tier (up to N times):

   Your previous response contained an invalid tool call. The error was: 
   missing required field "query" in function "search". 
   You must respond with valid JSON matching the tool schema exactly.
   Available tools: search, write_file, run_terminal_command.
   

2. After N retries, escalate to the next tier.
3. Log the failure for observability.

Configuration:

reliability:
  tool_validation:
    enabled: true
    max_retries: 3
    retry_with_correction: true   # inject corrective prompt on retry
    validate_json: true
    validate_schema: true
    validate_tool_names: true

Component 3: Token Budget Enforcement

Mechanism: Track token usage per session and per time window. Enforce configurable limits to prevent runaway agents from monopolizing the model server.

Limits:

• Per-session: Maximum tokens consumed in a single session (input + output)
• Per-hour: Maximum tokens across all sessions in a rolling hour
• Per-request output: Maximum output tokens for a single generation (prevents infinite reasoning loops)

On budget exceeded:

• warn_and_continue: Add a header X-MLX-Stack-Budget-Warning: 80% to responses
• escalate: Switch to a cheaper/faster tier for the remainder of the session
• hard_stop: Return an error and refuse further requests until the budget resets

Configuration:

reliability:
  token_budget:
    enabled: true
    per_session: 500000           # 500K tokens per session
    per_hour: 2000000             # 2M tokens per hour across all sessions
    max_output_tokens: 16384      # per-request output cap
    policy: warn_and_continue     # or: escalate, hard_stop

Component 4: Automatic Model Escalation

Mechanism: When a tier fails to produce a valid response (after retries), transparently re-route the request to the next tier in the escalation chain. The agent framework receives a working response and never knows the escalation happened.

Escalation chain:

fast → standard → longctx → premium (cloud via OpenRouter)

Implementation:

• On tool validation failure after max retries: escalate
• On loop detection circuit break: escalate
• On timeout (model too slow, possibly due to memory pressure): escalate
• On repeated 5xx errors from a tier: escalate and mark tier as degraded

Response headers for observability:

X-MLX-Stack-Tier: standard          # which tier actually served the request
X-MLX-Stack-Escalated-From: fast    # original tier before escalation
X-MLX-Stack-Escalation-Reason: tool_validation_failure
X-MLX-Stack-Retries: 3

Configuration:

reliability:
  escalation:
    enabled: true
    chain: [fast, standard, longctx, premium]
    max_retries_before_escalate: 3
    escalate_on:
      - tool_validation_failure
      - loop_detected
      - timeout
      - server_error
    cloud_fallback:
      provider: openrouter
      model: anthropic/claude-sonnet-4-6
      api_key_env: OPENROUTER_API_KEY

Component 5: Memory Pressure Quality Monitor (Bonus)

Mechanism: Monitor macOS memory pressure via sysctl vm.memory_pressure_level or the libdispatch memory pressure source. When the system enters memory pressure (indicating swap is active or imminent), model quality degrades silently — the model is technically serving responses but they're worse.

On memory pressure:

1. Add X-MLX-Stack-Memory-Pressure: warning header to all responses
2. If sustained for >60s, preemptively escalate complex requests to cloud
3. Log the event for postmortem analysis

Why This Belongs in mlx-stack (Not the Agent Framework)

1. Framework-agnostic: Every agent framework benefits. Hermes, CrewAI, OpenHands, AutoGPT — none of them need to implement their own reliability logic.
2. Model-aware: mlx-stack knows which tiers are available, their capabilities, and their health. The agent framework doesn't.
3. Transparent: The agent framework's code doesn't change. It sends requests and gets responses. The reliability layer is invisible when everything works.
4. Operational: Loop detection, token budgets, and memory pressure monitoring are infrastructure concerns, not application concerns.

Why This is a "WOW Factor" Feature

Today, the best advice on the CrewAI forum for dealing with unreliable local models is "keep trying different models until you find one that works." mlx-stack's reliability layer changes the value proposition from "hope your model is good enough" to "mlx-stack guarantees a working response through intelligent escalation."

The pitch becomes: 95% local (cheap, private, fast), 5% cloud (reliable safety net). Users get the economics and privacy of local inference with the reliability of cloud APIs.

Implementation Approach

Option A: LiteLLM custom callbacks (recommended)

LiteLLM supports custom callbacks that can intercept requests and responses. The reliability layer would be implemented as a callback class:

class ReliabilityCallback(litellm.Callbacks):
    async def async_post_call_success_hook(self, response, ...):
        # Loop detection
        # Tool call validation
        # Token budget tracking
        # Escalation decision

This avoids building a separate proxy and integrates directly into the existing LiteLLM process.

Option B: Standalone proxy

A thin FastAPI app that sits in front of LiteLLM, adding the reliability logic. This is more isolated but adds another process to manage.

Recommendation: Start with Option A (LiteLLM callbacks) for simplicity. Move to Option B only if the callback interface proves too limiting.

Priority

v0.2 — This solves the #1 reason people abandon local models for agent workloads. Combined with the existing multi-tier architecture and cloud fallback, it makes mlx-stack uniquely valuable in the local serving space.

Acceptance Criteria

• Loop detection identifies repeated identical/near-identical responses and triggers circuit breaker
• Tool call validation checks responses against provided function schemas
• Corrective retry injects helpful context before re-prompting the model
• Automatic escalation transparently re-routes failed requests to the next tier
• Cloud fallback works as the last resort in the escalation chain
• Token budget enforcement with configurable per-session and per-hour limits
• Response headers expose escalation metadata for observability
• All components individually configurable (enable/disable, thresholds)
• Memory pressure monitoring with proactive escalation
• Comprehensive logging of all reliability events
• Documentation explaining the reliability model and configuration
• Integration tests simulating common failure modes

[weklund]

Deep Analysis: Agent Reliability Layer (#29)

Is the User Pain Point Real? (Yes -- Extensively Documented)

Research strongly validates that unreliable tool calling and agent loops are the #1 blocker for local model adoption in agent workloads:

• How Do LLMs Fail In Agentic Scenarios? (Dec 2025) -- Analyzed 900 agentic executions across multiple LLMs using the KAMI benchmark. Found that tool-calling failures (malformed JSON, hallucinated tool names, missing required arguments) are among the most common and impactful failure modes. Smaller local models fail at significantly higher rates than cloud frontier models.

• Towards a Science of AI Agent Reliability (Feb 2026, Princeton) -- Formal framework decomposing agent reliability into consistency, robustness, and recovery dimensions. Key finding: rising benchmark accuracy scores do not predict real-world reliability -- agents that score well on benchmarks still fail in practice.

• The Multi-Agent Trap (Mar 2026) -- Reports that Google DeepMind found multi-agent networks amplify errors 17x. 40% of multi-agent projects get canceled due to reliability issues.

• 7 AI Agent Failure Modes (Galileo, Nov 2025) -- Documents the exact failure modes this issue describes: infinite loops, malformed tool calls, hallucinated function names, runaway token usage.

• Invariant Labs Loop Detection -- Open-source guardrail specifically for detecting and preventing infinite loops in agentic systems. Validates that this is a recognized infrastructure-level concern.

• r/AI_Agents -- Multiple highly-upvoted threads: "The Infinite Loop fear is real" (Jan 2026), "The $30K agent loop" (Dec 2025) describing real financial damage from uncontrolled agent loops.

Bottom line: The pain point is real, well-researched, and nobody is solving it at the serving layer. This is a genuine differentiator.

---

Open Questions / Decisions

#	Question	Why It Matters	Recommended Answer / PoC Needed
1	LiteLLM callback API vs. standalone proxy -- The issue proposes Option A (LiteLLM callbacks) vs Option B (standalone proxy). Which hooks are actually available?	Determines the entire architecture. If LiteLLM hooks cannot modify responses or re-route requests, Option A is dead.	LiteLLM hooks are sufficient. Confirmed from LiteLLM docs: async_pre_call_hook (modify/reject incoming requests), async_post_call_success_hook (modify outgoing responses), async_post_call_failure_hook (transform errors), async_post_call_response_headers_hook (inject custom headers). This covers all 5 components. PoC: write a minimal CustomLogger subclass that intercepts a response and logs it.
2	Session identity -- How does the reliability layer identify "the same session" across requests? The OpenAI API has no session concept.	Loop detection and token budgets require per-session state. Without session identity, you cannot track whether 3 consecutive responses are duplicates.	Options: (a) X-MLX-Stack-Session-Id custom header, (b) hash of the system prompt as implicit session key, (c) API key as session proxy. Need to decide which. Option (a) requires agent framework cooperation, (b) is zero-config but coarse, (c) is available today.
3	Corrective retry: who pays the latency? -- When a tool call fails validation and the reliability layer retries with a corrective prompt, the client is blocked waiting.	A corrective retry adds 2-5s of latency. For streaming responses, this is especially tricky -- do you buffer, or start streaming and then restart?	PoC needed: Measure retry latency on a real model. For streaming, the simplest approach is to buffer non-streaming retries and only stream the final successful attempt. Corrective retries should never be streamed mid-flight.
4	Escalation transparency vs. opacity -- When a request is escalated from fast to standard, should the client know?	Some agent frameworks may behave differently if they see a different model name. Others need to track which model responded for logging. Headers are non-breaking, but response body model name could confuse frameworks.	Recommend: Transparent via headers (X-MLX-Stack-Escalated-From), but preserve the original model name in the response body. Headers are opt-in observability; response body should look identical to what the client requested.
5	Similarity threshold for loop detection -- The issue proposes 0.95 fuzzy matching. What metric? Edit distance? Cosine similarity of embeddings?	The threshold directly controls false positives (legitimate repeated content flagged as loops) vs. false negatives (subtly different but semantically identical loops missed).	Recommend starting simple: normalize whitespace, strip timestamps/IDs, then exact string match on the normalized output. Fuzzy matching adds complexity and latency. A model stuck in a loop produces identical output, not merely similar output. PoC: collect 20 real loop examples from CrewAI/OpenHands logs and check if exact match catches them all.
6	LiteLLM already has fallbacks -- mlx-stack's litellm_gen.py already generates a fallback chain (fast -> standard -> longctx -> premium). Does the reliability layer duplicate this?	Risk of double-escalation: LiteLLM retries and the reliability layer retries simultaneously.	The reliability layer should replace LiteLLM's built-in fallback chain for agent workloads, not layer on top. When reliability mode is enabled, disable LiteLLM's general_settings.fallbacks and let the reliability layer own escalation decisions with richer context (loop state, tool validation, budget).
7	Memory pressure monitoring: macOS API availability -- The issue proposes sysctl vm.memory_pressure_level. Does this exist on all supported macOS versions?	If the API is not available or requires elevated permissions, this component is unbuildable.	PoC needed: Run sysctl -a | grep memory_pressure on macOS 14+ and verify. Alternative: use psutil.virtual_memory().percent which mlx-stack already depends on (used in stack_up.py for memory warnings).
8	Scope creep risk: 5 components at once -- The issue proposes loop detection, tool validation, token budgets, escalation, AND memory monitoring.	Shipping all 5 simultaneously increases risk and timeline. Some components (loop detection, escalation) are high-value and well-understood. Others (memory pressure quality monitoring) are speculative.	Recommend phased delivery: Phase 1 (loop detection + escalation) -> Phase 2 (tool validation + corrective retry) -> Phase 3 (token budgets) -> Phase 4 (memory pressure monitor).

---

Test Harness Strategy

Unit Tests (no models, no GPU)

Test	What to Assert	Approach
Loop detection: exact duplicate	3 identical responses trigger CIRCUIT_BREAK	Feed LoopDetector.check() with 3 copies of the same string, assert status
Loop detection: below threshold	2 identical + 1 different stays NORMAL	Feed 2 same + 1 different, assert NORMAL
Loop detection: window rollover	Old duplicates outside window do not count	Insert 3 duplicates spread across >window_size calls with different content between, assert no break
Loop detection: session isolation	Session A's loop does not affect session B	Interleave calls from two session IDs, assert only the looping session breaks
Tool validation: valid JSON + valid schema	Valid tool call passes validation	Provide tools schema, feed a correct {"name": "search", "arguments": {"query": "test"}}, assert PASS
Tool validation: malformed JSON	Non-JSON tool call triggers retry	Feed {"name": "search", "arguments": {query: test}} (invalid JSON), assert FAIL
Tool validation: hallucinated tool name	Unknown function name triggers retry	Feed {"name": "execute_code", ...} when tools only has run_terminal_command, assert FAIL
Tool validation: missing required args	Required field absent triggers retry	Feed {"name": "search", "arguments": {}} when query is required, assert FAIL
Token budget: under limit	Requests under budget proceed normally	Track 100K tokens in a 500K budget session, assert NORMAL
Token budget: over limit	Exceeding per-session limit triggers policy	Track 600K tokens in a 500K budget, assert policy action fired
Token budget: hourly rolling window	Budget resets after window expires	Set 1-hour window, mock time advancing past window, assert budget resets
Escalation chain: ordered	Escalation follows configured chain order	Configure chain [fast, standard, premium], trigger escalation twice, assert fast->standard->premium
Escalation chain: terminal	Last tier returns error, not infinite escalation	Exhaust all tiers, assert structured error returned
Corrective prompt injection	Retry includes error context in system message	Trigger tool validation failure, capture the retried request, assert corrective message present
Response header injection	Escalation metadata appears in headers	Mock an escalated response, assert X-MLX-Stack-Tier and X-MLX-Stack-Escalated-From headers
Config parsing	All reliability config options parsed correctly	Load a sample YAML with all options, assert each field maps to the correct internal config
Config defaults	Missing config fields use documented defaults	Load minimal YAML, assert defaults for window_size, threshold, etc.
Disabled components	enabled: false completely bypasses a component	Set loop_detection.enabled: false, feed duplicates, assert no circuit break

Integration Tests (require real LiteLLM proxy + mock model)

Test	What to Assert
test_loop_breaks_circuit	Send 5 identical chat completions through the proxy; assert the 3rd-5th return a loop_detected error or are escalated
test_tool_validation_corrective_retry	Send a request with tools defined; mock vllm-mlx to return malformed JSON on first call, valid JSON on second; assert client receives valid response and X-MLX-Stack-Retries: 1 header
test_escalation_transparent_to_client	Configure fast+standard tiers; mock fast to always return 500; send request; assert response comes from standard tier with escalation headers
test_token_budget_enforcement	Set per-session budget to 1000 tokens; send requests until budget exceeded; assert appropriate policy action (warning header or error)
test_streaming_compatibility	Send a streaming request that triggers a corrective retry; assert the client receives a valid SSE stream (not a mix of failed + successful streams)
test_reliability_headers_always_present	Send a normal (non-escalated) request; assert X-MLX-Stack-Tier header is present even on success
test_config_hot_reload	Change reliability config while proxy is running; assert new thresholds take effect without restart

Benchmark / Acceptance Test

@pytest.mark.benchmark
def test_reliability_overhead(benchmark_runner):
    """Measure latency overhead of the reliability layer.
    With reliability enabled vs. disabled, assert <5ms overhead
    on the hot path (no loop, no validation failure)."""
    results_enabled = benchmark_runner.run(reliability=True, n_requests=100)
    results_disabled = benchmark_runner.run(reliability=False, n_requests=100)
    overhead = results_enabled.mean_latency - results_disabled.mean_latency
    assert overhead < 5  # ms -- reliability layer should be nearly free on happy path

---

Prior Art / Existing Solutions

Tool	What It Does	Gap for mlx-stack
LiteLLM built-in fallbacks	Retries and routes to fallback models on HTTP errors	No semantic awareness -- does not detect loops, malformed tool calls, or quality degradation
Invariant Labs Explorer	Agent trace analysis and loop detection guardrails	Post-hoc analysis tool, not real-time serving-layer enforcement
NeMo Guardrails	Input/output filtering for safety	Focused on content safety, not agent reliability (loops, tool validation)
NeuralTrust Circuit Breakers	Representation engineering to hardwire safety	Research technique modifying model weights, not applicable at serving layer
LiteLLM Custom Guardrails	Framework for custom pre/post-call logic	This is the implementation vehicle, not a competing solution. The reliability layer would be built as a LiteLLM custom guardrail/callback.

Nobody is doing serving-layer agent reliability with automatic model escalation. This is genuinely novel.

---

Recommendation

1. The user problem is validated by multiple peer-reviewed papers and widespread community reports. This is worth building.
2. LiteLLM's callback API is the right implementation vehicle -- it has all the hooks needed (pre-call, post-call, response modification, custom headers, error transformation).
3. Phase the delivery: Start with loop detection + escalation (highest impact, well-understood), then add tool validation, then token budgets.
4. The 8 open questions above need answers before implementation begins -- particularly session identity (fix: use huggingface_hub Python API for model pull #2), retry latency (Handle gated HuggingFace models gracefully without owning credential management #3), and the interaction with existing LiteLLM fallbacks (fix: add [proxy] extra to litellm install and bump to post-CVE v1.83.0 #6).
5. Estimated effort (revised): Phase 1 (loop detection + escalation): ~2 weeks. Phase 2 (tool validation + corrective retry): ~2 weeks. Full feature: ~4-6 weeks.