Skip to content

models.md

Latest commit

 

History

History
380 lines (321 loc) · 12.2 KB

models.md

File metadata and controls

380 lines (321 loc) · 12.2 KB
Raw

Custom Models

Add custom providers and models (Ollama, vLLM, LM Studio, proxies) via ~/.dreb/agent/models.json.

Table of Contents

Minimal Example

For local models (Ollama, LM Studio, vLLM), only id is required per model:

{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "api": "openai-completions",
      "apiKey": "ollama",
      "models": [
        { "id": "llama3.1:8b" },
        { "id": "qwen2.5-coder:7b" }
      ]
    }
  }
}

The apiKey is required but Ollama ignores it, so any value works.

Some OpenAI-compatible servers do not understand the developer role used for reasoning-capable models. For those providers, set compat.supportsDeveloperRole to false so dreb sends the system prompt as a system message instead. If the server also does not support reasoning_effort, set compat.supportsReasoningEffort to false too.

You can set compat at the provider level to apply to all models, or at the model level to override a specific model. This commonly applies to Ollama, vLLM, SGLang, and similar OpenAI-compatible servers.

{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "api": "openai-completions",
      "apiKey": "ollama",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false
      },
      "models": [
        {
          "id": "gpt-oss:20b",
          "reasoning": true
        }
      ]
    }
  }
}

Full Example

Override defaults when you need specific values:

{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "api": "openai-completions",
      "apiKey": "ollama",
      "models": [
        {
          "id": "llama3.1:8b",
          "name": "Llama 3.1 8B (Local)",
          "reasoning": false,
          "input": ["text"],
          "contextWindow": 128000,
          "maxTokens": 32000,
          "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
        }
      ]
    }
  }
}

The file reloads each time you open /model. Edit during session; no restart needed.

Supported APIs

API Description
openai-completions OpenAI Chat Completions (most compatible)
openai-responses OpenAI Responses API
anthropic-messages Anthropic Messages API
google-generative-ai Google Generative AI

Set api at provider level (default for all models) or model level (override per model).

Provider Configuration

Field Description
baseUrl API endpoint URL
api API type (see above)
apiKey API key (see value resolution below)
headers Custom headers (see value resolution below)
authHeader Set true to use the resolved apiKey as Authorization: Bearer instead of provider API-key auth
models Array of model configurations
modelOverrides Per-model overrides for built-in models on this provider

Value Resolution

The apiKey and headers fields support three formats:

  • Shell command: "!command" executes and uses stdout 
    "apiKey": "!security find-generic-password -ws 'anthropic'"
"apiKey": "!op read 'op://vault/item/credential'"
  • Environment variable: Uses the value of the named variable 
    "apiKey": "MY_API_KEY"
  • Literal value: Used directly 
    "apiKey": "sk-..."

Bearer Auth for Anthropic-Compatible Providers

Third-party Anthropic-compatible endpoints use x-api-key by default. If an endpoint instead requires Authorization: Bearer <key>, set authHeader: true:

{
  "providers": {
    "company-anthropic": {
      "baseUrl": "https://ai.example.com/anthropic",
      "api": "anthropic-messages",
      "apiKey": "COMPANY_ANTHROPIC_TOKEN",
      "authHeader": true,
      "models": [
        { "id": "company-claude" }
      ]
    }
  }
}

For the built-in anthropic-messages implementation, this selects Bearer-only auth: dreb sends the request-time resolved credential as Authorization and does not also send x-api-key. The credential can use any value resolution format; the environment variable does not need a special Anthropic SDK name.

The flag also works when redirecting a built-in provider without redefining its models:

{
  "providers": {
    "anthropic": {
      "baseUrl": "https://ai.example.com/anthropic",
      "apiKey": "COMPANY_ANTHROPIC_TOKEN",
      "authHeader": true
    }
  }
}

Leave authHeader unset or false for endpoints that expect x-api-key.

Custom Headers

{
  "providers": {
    "custom-proxy": {
      "baseUrl": "https://proxy.example.com/v1",
      "apiKey": "MY_API_KEY",
      "api": "anthropic-messages",
      "headers": {
        "x-portkey-api-key": "PORTKEY_API_KEY",
        "x-secret": "!op read 'op://vault/item/secret'"
      },
      "models": [...]
    }
  }
}

Model Configuration

Field Required Default Description
id Yes — Model identifier (passed to the API)
name No id Human-readable model label. Used for matching (--model patterns) and shown in model details/status text.
api No provider's api Override provider's API for this model
reasoning No false Supports extended thinking
input No ["text"] Input types: ["text"] or ["text", "image"]
contextWindow No 128000 Context window size in tokens
maxTokens No 16384 Maximum output tokens
cost No all zeros {"input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0} (per million tokens)
compat No provider compat OpenAI compatibility overrides. Merged with provider-level compat when both are set.

Current behavior:

  • /model and --list-models list entries by model id.
  • The configured name is used for model matching and detail/status text.

Overriding Built-in Providers

Route a built-in provider through a proxy without redefining models:

{
  "providers": {
    "anthropic": {
      "baseUrl": "https://my-proxy.example.com/v1"
    }
  }
}

All built-in Anthropic models remain available. API key auth continues to work (ANTHROPIC_API_KEY).

To merge custom models into a built-in provider, include the models array:

{
  "providers": {
    "anthropic": {
      "baseUrl": "https://my-proxy.example.com/v1",
      "apiKey": "ANTHROPIC_API_KEY",
      "api": "anthropic-messages",
      "models": [...]
    }
  }
}

Merge semantics:

  • Built-in models are kept.
  • Custom models are upserted by id within the provider.
  • If a custom model id matches a built-in model id, the custom model replaces that built-in model.
  • If a custom model id is new, it is added alongside built-in models.

Per-model Overrides

Use modelOverrides to customize specific built-in models without replacing the provider's full model list.

{
  "providers": {
    "openrouter": {
      "modelOverrides": {
        "anthropic/claude-sonnet-4": {
          "name": "Claude Sonnet 4 (Bedrock Route)",
          "compat": {
            "openRouterRouting": {
              "only": ["amazon-bedrock"]
            }
          }
        }
      }
    }
  }
}

modelOverrides supports these fields per model: name, reasoning, input, cost (partial), contextWindow, maxTokens, headers, compat.

Behavior notes:

  • modelOverrides are applied to built-in provider models.
  • Unknown model IDs are ignored.
  • You can combine provider-level baseUrl/headers with modelOverrides.
  • If models is also defined for a provider, custom models are merged after built-in overrides. A custom model with the same id replaces the overridden built-in model entry.

Reasoning Across Model Switches

A custom model's provider identity is part of reasoning-state compatibility; matching endpoint URLs or model IDs alone is not enough. Exact-model signed, encrypted, or redacted reasoning state is replayed unchanged. Between different models, structured reasoning is preserved only when both models use the same provider and openai-completions API, the destination accepts structured reasoning, and the source uses a recognized plain field: reasoning_content, reasoning, or reasoning_text.

For other targets, readable reasoning is retained as labelled plaintext inside <reformatted-pre-switch-reasoning> markers after incompatible protocol metadata is stripped. Redacted or encrypted-only opaque state is omitted. This conversion happens only for the outbound request and does not alter session history, so returning to the original model can replay its original state unless history has been compacted or pruned. Portability also depends on compatible provider signatures.

OpenAI Compatibility

For providers with partial OpenAI compatibility, use the compat field.

  • Provider-level compat applies defaults to all models under that provider.
  • Model-level compat overrides provider-level values for that model.
{
  "providers": {
    "local-llm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "compat": {
        "supportsUsageInStreaming": false,
        "maxTokensField": "max_tokens"
      },
      "models": [...]
    }
  }
}
Field Description
supportsStore Provider supports store field
supportsDeveloperRole Use developer vs system role
supportsReasoningEffort Support for reasoning_effort parameter
reasoningEffortMap Map dreb thinking levels to provider-specific reasoning_effort values
supportsUsageInStreaming Supports stream_options: { include_usage: true } (default: true)
maxTokensField Use max_completion_tokens or max_tokens
requiresToolResultName Include name on tool result messages
requiresAssistantAfterToolResult Insert an assistant message before a user message after tool results
requiresThinkingAsText Convert thinking blocks to plain text
thinkingFormat Use reasoning_effort, reasoning: { effort }, zai, qwen, qwen-chat-template, or nested thinking: { type, effort? } parameters
supportsStrictMode Include the strict field in tool definitions
openRouterRouting OpenRouter routing config passed to OpenRouter for model/provider selection
vercelGatewayRouting Vercel AI Gateway routing config for provider selection (only, order)

qwen uses top-level enable_thinking. Use qwen-chat-template for local Qwen-compatible servers that require chat_template_kwargs.enable_thinking.

Example:

{
  "providers": {
    "openrouter": {
      "baseUrl": "https://openrouter.ai/api/v1",
      "apiKey": "OPENROUTER_API_KEY",
      "api": "openai-completions",
      "models": [
        {
          "id": "openrouter/anthropic/claude-3.5-sonnet",
          "name": "OpenRouter Claude 3.5 Sonnet",
          "compat": {
            "openRouterRouting": {
              "order": ["anthropic"],
              "fallbacks": ["openai"]
            }
          }
        }
      ]
    }
  }
}

Vercel AI Gateway example:

{
  "providers": {
    "vercel-ai-gateway": {
      "baseUrl": "https://ai-gateway.vercel.sh/v1",
      "apiKey": "AI_GATEWAY_API_KEY",
      "api": "openai-completions",
      "models": [
        {
          "id": "moonshotai/kimi-k2.5",
          "name": "Kimi K2.5 (Fireworks via Vercel)",
          "reasoning": true,
          "input": ["text", "image"],
          "cost": { "input": 0.6, "output": 3, "cacheRead": 0, "cacheWrite": 0 },
          "contextWindow": 262144,
          "maxTokens": 262144,
          "compat": {
            "vercelGatewayRouting": {
              "only": ["fireworks", "novita"],
              "order": ["fireworks", "novita"]
            }
          }
        }
      ]
    }
  }
}