Native-first proxy (0.7.3)

[antoinezambelli]

Native-first proxy (0.7.3)

With native function calling now well-supported across modern local models, this release makes the OpenAI-compatible proxy native-first: by default it forwards the client's OpenAI tools / messages to the backend verbatim and is optimized for that path. Prompt-injection is preserved as an explicit opt-in for llama.cpp / llamafile backends that lack a function-calling template — but it is no longer the default.

This branch also folds in the commits that landed on main since 0.7.2 so they ship under one version (the OpenAICompatClient, the configurable backend timeout, and several proxy/eval fixes).

⚠️ Breaking change

The proxy --mode {native,prompt} flag is renamed to --backend-capability {native,prompt} (and ProxyServer(mode=…) → ProxyServer(backend_capability=…)).

• --mode collided with the proxy's existing managed / external deployment mode. The new name states what it actually controls — the backend's tool-calling protocol — and reflects that the choice is declared once at startup and frozen, never probed or switched mid-request.
• No deprecation alias. --mode was only introduced in 0.7.1, so it is removed cleanly rather than carried as a confusingly-named compatibility shim.
• Migration: --mode native → drop it (native is the default) or --backend-capability native; --mode prompt → --backend-capability prompt.

A note on the prompt path

Prompt-injection remains fully supported as --backend-capability prompt, and it's the right tool for a backend with no function-calling template. But whether a model stays coherent across multi-turn tool results in prompt mode varies by model, and it tends to degrade on more complex, multi-step agentic interactions. Treat it as a fallback for non-FC backends, not the recommended path — reach for native whenever the backend supports it.

What's included

Added

• OpenAICompatClient for arbitrary OpenAI-compatible endpoints. Add OpenAICompatClient for OpenAI-compatible endpoints #89 (thanks @lucasgerads).
• --backend-timeout proxy option — configurable backend response timeout (default 300s). Add proxy backend timeout option #91.
• --backend-capability {native,prompt} proxy flag — native (default) forwards tools/messages verbatim to a function-calling-capable backend; prompt opts into prompt-injection for non-FC llama.cpp/llamafile backends. Declared once at startup and frozen.
• Effective backend timeout logged at proxy startup.

Changed

• BREAKING — --mode → --backend-capability (see above).
• Native function calling is now transparent passthrough — the proxy forwards the client's OpenAI tool/message payloads verbatim instead of round-tripping them through forge's internal ToolSpec representation, which dropped schema detail.
• vLLM model identity consolidated to a single source of truth (the wire model_path and the registry model key are set together). Extract VLLMClient model-field derivation into a helper to avoid post-construction mutation #75.
• The prompt capability is now rejected loudly for ollama / vllm / anthropic backends — previously it was silently ignored for ollama.
• stream_options is excluded from proxy passthrough. fix: exclude stream_options from proxy passthrough #94 (thanks @alexandergunnarson).

Fixed

• Consistent malformed-tool-call / unexpected-response handling across the OpenAI-shape clients — malformed model tool args drive a retry (TextResponse) instead of degrading silently or raising inconsistently, and non-streaming responses are guarded so a broken provider envelope fails loud.
• Guardrails.record() no longer drops tool args for prerequisite tracking. Fix Guardrails.record() dropping tool args for prerequisite tracking #72 (thanks @hobostay).
• Deprecated asyncio API replaced; proxy server input validation added. Fix deprecated asyncio API and add input validation in proxy server #71 (thanks @hobostay).
• Proxy input hardening, non-blocking Ollama stop, client shutdown, and loud arg decode. Post-review fixes: proxy input hardening, non-blocking Ollama stop, client shutdown, loud arg decode #86.
• Dead code and a fragile variable reference cleaned up in LlamafileClient. Clean up dead code and fragile variable reference in LlamafileClient #73 (thanks @hobostay).

Removed

• Runtime auto function-calling mode in LlamafileClient — the proxy never used it, and its mid-request probe-and-switch is replaced by the declared-and-frozen --backend-capability.

Validation

• Unit suite: 1086 passed.
• Regression eval against the 0.7.0 baseline (single reference model, llama.cpp server, reforged, N=50 across 8 scenarios spanning the changed surfaces): both native and prompt track the baseline within sampling noise — no behavioral regression from the rewrite or the malformed-response hardening.
• The native-first / verbatim-passthrough design is documented in ADR-012; backend setup and the user guide are updated for --backend-capability and the prompt-path caveat.