Custom (EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM) Model Provider Services are unusable: no tool routes to 'custom', and pi has no --provider

[FantomKun]

Summary

A Unity Catalog Model Provider Service with provider_type: EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM — a self-hosted, OpenAI-compatible model registered in the AI Gateway — cannot be used by any ucode agent. It never appears in ucode configure, and there is no flag to select it, so a model that the gateway itself routes correctly is unreachable through ucode.

The motivation is running a private model (in our case DeepSeek-V4-Flash on SGLang, reached over PrivateLink/NCC) with governance, rate limits, and inference tables handled by the gateway, exactly like a vendor-backed service — but consumed from a coding agent.

Two independent gaps, both in ucode rather than the gateway.

1. No tool can route to a custom provider type.

src/ucode/databricks.py (_TOOL_PROVIDER_TYPES, ~L1473):

_TOOL_PROVIDER_TYPES: dict[str, tuple[str, ...]] = {
    "claude": ("anthropic", "amazon_bedrock"),
    "codex": ("openai",),
}

custom appears nowhere, and there is no pi key. tool_supports_provider_type("pi", "custom") is False, so resolve_provider_service rejects the service:

Model provider service 'llm_gov.gateway.deepseek-v4-flash' is a 'custom' provider,
which pi can't route to (supported: none).

Separately, only codex_cmd and claude_cmd declare a --provider option — pi_cmd has none — and _maybe_select_provider_service early-returns for anything that isn't claude/codex, so the interactive picker never offers one either.

2. agents/pi.py has no provider path.

configure_tool (src/ucode/agents/__init__.py ~L334) still reads # provider routing is claude/codex-only; every other tool needs a model, pi.write_tool_config takes no provider, and render_overlay only ever emits the three fixed databricks-claude/-openai/-gemini providers.

Reproduction

Against a workspace with a custom Model Provider Service (verified 2026-08-06):

$ databricks api get /api/2.1/unity-catalog/model-provider-services --profile <P>

{
  "name": "model-provider-services/llm_gov.gateway.deepseek-v4-flash",
  "config": {
    "provider_type": "EXTERNAL_MODEL_PROVIDER_TYPE_CUSTOM",
    "targets": [{"model": "deepseek-v4-flash",
                 "native_api_types": ["openai/v1/chat/completions"]}]
  }
}

from ucode.databricks import resolve_provider_service, list_tool_provider_services
resolve_provider_service("pi", "llm_gov.gateway.deepseek-v4-flash", ws, token)
# -> (None, "... is a 'custom' provider, which pi can't route to (supported: none).")
list_tool_provider_services("pi", ws, token)   # -> ([], None)  — never offered

There is also no flag to ask for one. Because pi_cmd is registered with ignore_unknown_options, --provider is not rejected — it's silently forwarded to the pi binary, which ignores it too, so the launch quietly proceeds on a Databricks model:

$ ucode pi --provider llm_gov.gateway.deepseek-v4-flash
  Model: system.ai.claude-opus-5      # provider silently ignored, no warning

Meanwhile the gateway routes it fine. Note the shape — the service is selected by header, and the body's model is the bare target name:

curl https://<ws>/ai-gateway/openai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -H "Databricks-Model-Provider-Service: llm_gov.gateway.deepseek-v4-flash" \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"say hi"}],"max_tokens":20}'
# -> HTTP 200, normal completion

Both halves matter. Passing the fully-qualified service name as model fails even with the header set:

{"error_code":"PERMISSION_DENIED","message":"Model 'llm_gov.gateway.deepseek-v4-flash'
 is not in the allowed models list for '...'"}

and dropping the header, keeping the bare target name, fails too:

{"error_code":"NOT_FOUND","message":"'deepseek-v4-flash' does not exist."}

Also worth noting build_tool_base_url has no builder for /ai-gateway/openai/v1, and build_pi_base_urls only builds claude/codex/gemini.

Verified working on that path: streaming, tool/function calling with streamed tool_calls deltas, the tool-result round trip, stream_options.include_usage, strict-mode tool schemas, and reasoning_effort ("none" suppresses reasoning; minimal–high return reasoning_content).

Third gap, found while fixing: native_api_types is discarded

_provider_service_entry parses targets[].model but drops targets[].native_api_types. That field is the only signal for which request dialect a target serves, and therefore which gateway path and client api type to use. Without it there's no way to tell a chat-completions target from a openai/v1/responses one except by guessing — which turns a clear configure-time error into a 404 mid-session.

Suggested fix

Keep the routing knowledge dialect-level and agent-agnostic, then land one agent:

1. databricks.py: retain native_api_types per target; add a dialect→gateway-path map (openai/v1/chat/completions → /ai-gateway/openai/v1) with a build_native_api_base_url helper; add CUSTOM_PROVIDER_TYPES and a custom_openai_chat_targets selector; extend service_usable_for_tool / resolve_provider_service so a custom service with no routable target is rejected with an actionable message rather than surfacing in the picker.
2. agents/pi.py: emit a databricks-custom provider (api: openai-completions) carrying the Databricks-Model-Provider-Service header, with the targets as its models. Pi is a good first landing because its models.json can express an arbitrary OpenAI-compatible provider declaratively.
3. cli.py: --provider on ucode pi (the option is byte-identical across commands and can be extracted to a shared Annotated), and include pi in _maybe_select_provider_service.

Two details worth calling out for whoever picks this up:

A latent bug in pi's token refresh. _refresh_token_once raises RuntimeError when default_model(state) is None, and _refresh_forever swallows that exception. Under a provider-only launch the workspace may expose no Databricks model at all (the service's targets are the models), so the background thread silently stops refreshing and the session dies when the token expires (~1h). This needs handling in any provider-for-pi change.

Only three compat flags are needed for an unknown OpenAI-compatible backend: maxTokensField: "max_tokens", supportsDeveloperRole: false, supportsStore: false. Pi's detectCompat already returns the right defaults for supportsReasoningEffort, supportsUsageInStreaming and supportsStrictMode on an unrecognized base URL. (A caution for anyone hand-writing this config: compat.thinkingFormat has a closed enum — openai | openrouter | together | deepseek | zai | qwen | chat-template | qwen-chat-template | string-thinking | ant-ling. A value like "reasoning_effort" is silently accepted, because ProviderCompatSchema is a loose union with no additionalProperties, but matches no branch at runtime and is a no-op.)

The context-window problem (a request, not just a fix)

The Model Provider Service API exposes no context-window metadata. Dumping every config key on our service gives only allow_all_targets, custom, forward_headers, forward_query_parameters, forward_unmanaged_paths, inference_table, provider_type, targets. Nothing about context length or max output tokens, and nothing per-target beyond model and native_api_types.

That leaves any client guessing, and the guess is not safe in both directions. With pi specifically:

• Omitting contextWindow defaults to 128000 (provider-composer.js).
• Compaction triggers at contextTokens > contextWindow - reserveTokens (default reserve 16384).
• On a server-side overflow pi compacts and retries once, compacting to contextWindow - reserveTokens. If the declared window is above the server's real limit, the retry overflows again and the turn ends with "Context overflow recovery failed after one compact-and-retry attempt."
• clampMaxTokensToContext also sizes per-request max_tokens from the declared window.

So understating degrades gracefully (earlier compaction, shorter replies) while overstating fails unrecoverably. A conservative default plus an explicit override is the only safe client-side answer.

Concrete evidence that a hardcoded guess goes stale: this same endpoint reported a 32768-token limit when first measured, and 327680 when re-probed a day later — the server had been resized, with no way for a client to notice. Probing is the only way to learn the real value, and it costs a deliberately oversized request:

The input (400005 tokens) is longer than the model's context length (327680 tokens).

Request: expose context window / max output tokens on the Model Provider Service target metadata. Until then, clients can only guess conservatively and offer a manual override.

Not a duplicate of #234

#234 covers FMAPI-backed UC model services in user schemas, blocked by the client-side _MODEL_SERVICE_REQUIRED_PREFIX = "system.ai." filter, where the model id goes in the request body. This issue is the model provider services API (/api/2.1/unity-catalog/model-provider-services), where routing is by header with a bare target name, and the blocker is the provider-type table plus a missing agent config path. Different API, different failure, non-overlapping fixes — though both point at the same theme of custom models being second-class in discovery. Adjacent: #224 (--model pin for custom UC model services), #204 (pi model detection).

I have a working patch for the above against main, including a tests/test_e2e.py case that passes against a live custom service. Verified end to end: discovery, resolve, generated config, a real tool round-trip through pi, token refresh with zero Databricks models on the workspace, the context-window override, and cleanup leaving a hand-added provider intact. Happy to open a PR if the approach looks right.

One note for CI: a custom service fronts a caller-operated endpoint, which can be down independently of ucode — the gateway surfaces that as a 502 wrapping the upstream's 503. The e2e test treats that as a skip rather than a failure (mirroring the existing USE CONNECTION/EXECUTE permission skip), since it isn't a ucode defect.