Feature: let vision/web-search sidecar use any configured provider's model (drop openai|anthropic-only backend)

[Lsj17]

Area

Proxy and routing · Dashboard · Catalog / models

What are you trying to accomplish?

I want to pick my own provider's model as the vision sidecar and web-search sidecar, from the same dropdowns where I select models — not be forced to use a native ChatGPT/Claude model. For example: route the main model through z.ai GLM / DeepSeek / Kimi / Qwen, and have the vision sidecar describe images with Qwen-VL (or whatever vision model I have quota on), and the web-search sidecar execute searches with a model from a provider I actually have available, instead of borrowing the ChatGPT or Anthropic account.

What prevents this today?

The sidecar backend is hardcoded to "openai" | "anthropic" and the model must be a native ChatGPT model:

• src/types.ts: OcxVisionSidecarConfig.backend?: "openai" | "anthropic" and OcxWebSearchSidecarConfig.backend?: "openai" | "anthropic" (same in OcxClaudeDesktopProfile.webSearchSidecar/visionSidecar).
• The management API rejects any other value with 400: src/server/management/config-routes.ts — "webSearch.backend must be openai, anthropic, or null" / "vision.backend must be openai, anthropic, or null".
• src/web-search/index.ts resolveSidecarBackend() and src/vision/index.ts resolveVisionBackend() only resolve to "openai" / "anthropic".

So in the UI dropdowns the sidecar model list only shows native ChatGPT models, and there is no provider: selector to route the sidecar through any other configured provider.

What should OpenCodex do?

Allow the sidecar backend to reference any configured provider (by provider id), with model set to any model that provider exposes:

• backend accepts a provider id (e.g. "zai", "deepseek", "moonshot", "qwen", ...) in addition to "openai" | "anthropic".
• The vision/search sidecar dropdowns in the UI list models from all configured providers (with a capability hint: vision-capable for the vision sidecar), not just ChatGPT natives.
• If the sidecar model is a text-only model, the existing graceful-degradation contract (e.g. [image omitted: …] marker) still applies.

Example usage or interface

// ocx config (sidecar settings)
{
  "visionSidecar": {
    "backend": "qwen",            // any configured provider id, not just openai|anthropic
    "model": "qwen3-vl-plus"      // model from that provider's catalog
  },
  "webSearchSidecar": {
    "backend": "zai",
    "model": "glm-5.2"
  }
}

In the Dashboard sidecar settings, the backend/model dropdown would show e.g. openai · anthropic · zai · deepseek · moonshot · qwen … and the model list follows the selected provider.

Alternatives or workarounds

• feat(sidecar): add Exa and other search providers as web-search sidecar backends #414 proposes adding third-party search APIs (Exa) as extra hardcoded backends — helpful, but it does not let me point the sidecar at a provider I already have configured and funded.
• Today I can only work around this by maintaining a ChatGPT/Claude account purely as the sidecar backend, even when my main provider has its own capable vision model.

Additional context

Related but not identical existing issues:

• [Bug] Web Search/Vision Sidecar Backend Fixed to OpenAI·Anthropic - Routing Turns Fail with 499/502 When Limits are Exhausted #398 — confirmed the openai|anthropic-only constraint (as a bug source), focused on 499/502 failures when those accounts are exhausted.
• feat:Enable DeepSeek and other text-only input models to perform image recognition and search functions. #88 — vision sidecar for text-only models was implemented, but the description model stays a native ChatGPT model (visionSidecar.model, default gpt-5.2).
• [Bug] Custom-provider vision capability is ambiguous when model metadata is missing #1024 — custom-provider vision capability ambiguity (registry metadata missing), different angle.
• [Feature]: expose remaining vision sidecar controls in Dashboard #1201 — exposing sidecar config in the web UI, related to the dropdown surface.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[Wibias]

Closing this as superseded by the existing split work, rather than tracking vision and web-search provider support together in one umbrella issue.

The two sidecars have different capability and transport requirements, so keeping them under one generic "any configured provider" abstraction would blur separate implementation problems.

Vision sidecar

The vision part is already covered by:

• feat(vision): add chat and Google sidecars #1161 — adds OpenAI-compatible/chat and Google vision sidecar backends, namespaced provider-model resolution, and related dashboard controls.
• [Feature]: expose remaining vision sidecar controls in Dashboard #1201 — tracks the remaining dashboard/UI surface for configuring the vision sidecar.
• [Bug] Custom-provider vision capability is ambiguous when model metadata is missing #1024 — tracks the separate problem of determining vision capability for custom-provider models when authoritative modality metadata is missing.

Web-search sidecar

The web-search part is already split by actual search capability:

• feat(sidecar): add Exa and other search providers as web-search sidecar backends #414 — dedicated search APIs such as Exa.
• feat(sidecar): investigate search-API-capable providers (e.g. Gemini) as web-search sidecar backends #415 — provider-native search capabilities such as Gemini Grounding / google_search.

An arbitrary configured LLM cannot automatically act as a web-search backend. The provider needs either a native search/tool API or a dedicated search service integration, so this should stay tracked through #414/#415 rather than through a generic provider/model selector.

The original goal of #1276 is still valid: users should not be forced to keep an OpenAI or Anthropic account only to provide auxiliary vision/search functionality. The implementation work is simply better tracked in the narrower issues above.

Closing #1276 as superseded by #1161, #1201, #1024, #414, and #415.