Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/filters/llmisvc_model_provider_resolver.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
<!-- Generated by: cargo xtask generate-filter-docs -->
<!-- Do not edit manually -->

# `llmisvc_model_provider_resolver`

Ports the `LLMISvc` / `KServe` BBR body-rewrite branch from IPP's `model-provider-resolver`.

## Configuration Notes

Reads the model name from the configured request header (default `X-Model`, typically set by an earlier `model_to_header`). When that value is a publisher ID (`publishers/.../models/<name>`), rewrite the body `"model"` field to `<name>` only. The routing header is never modified -- `KServe` routes on the publisher ID.

If the header is absent/empty, or the body has no `"model"` field, this filter is a no-op (it does not invent a body `"model"`).

Does **not** resolve `ExternalModel` / `ExternalProvider` CRDs, perform weighted provider selection, rewrite `Host`, or inject credentials.

## Configuration

| Field | Type | Required | Description |
|-------|------|---------|-------------|
| `header` | string | no | Request header that carries the publisher ID for `KServe` routing. Defaults to `X-Model` (same as `model_to_header`). |
| `max_body_bytes` | integer | no | Maximum request body size to buffer before parsing. |

## Example

```yaml
filter: llmisvc_model_provider_resolver
header: X-Model # optional, defaults to X-Model
```
1 change: 1 addition & 0 deletions docs/filters/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ see the [Praxis core filter reference][core-ref].

| Filter | Description |
|--------|-------------|
| [`llmisvc_model_provider_resolver`](llmisvc_model_provider_resolver.md) | Ports the `LLMISvc` / `KServe` BBR body-rewrite branch from IPP's `model-provider-resolver`. |
| [`model_to_header`](model_to_header.md) | Promotes the JSON `"model"` field from the request body to a request header. |

### Prompt Enrich
Expand Down
1 change: 1 addition & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ before sending requests.
| [intelligent-route-mcp.yaml](configs/intelligent-route-mcp.yaml) | Routes MCP `tools/call` requests to the cluster that owns the requested tool, using the `mcp.name` metadata set by the `mcp` filter |
| [intelligent-route-overlay.yaml](configs/intelligent-route-overlay.yaml) | Routes requests using a routing overlay file (`routing-overlay.json`) instead of inline YAML candidates |
| [json-rpc-routing.yaml](configs/json-rpc-routing.yaml) | Routes JSON-RPC 2.0 requests to different backends based on the "method" field in the JSON request body |
| [llmisvc-model-provider-resolver.yaml](configs/llmisvc-model-provider-resolver.yaml) | Ports the LLMISvc / KServe BBR body-rewrite path: when the model name is a publisher ID (`publishers/{ns}/models/{name}`), rewrite the JSON body `"model"` field to `{name}` so vLLM receives the short name |
| [mcp-classifier-routing.yaml](configs/mcp-classifier-routing.yaml) | Routes MCP requests by body-derived method and tool name |
| [mcp-stateless-broker.yaml](configs/mcp-stateless-broker.yaml) | Configurable stateless MCP broker using the final MCP 2026-07-28 stateless profile |
| [model-to-header-routing.yaml](configs/model-to-header-routing.yaml) | Routes LLM API requests to different backends based on the "model" field in the JSON request body |
Expand Down
46 changes: 46 additions & 0 deletions examples/configs/llmisvc-model-provider-resolver.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# LLMISvc Model Provider Resolver
#
# Build:
# cargo build -p praxis-ai-proxy
#
# Ports the LLMISvc / KServe BBR body-rewrite path: when the model
# name is a publisher ID (`publishers/{ns}/models/{name}`), rewrite
# the JSON body `"model"` field to `{name}` so vLLM receives the
# short name. The routing header (default `X-Model`) is left
# unchanged so KServe can still route on the publisher ID.
#
# Typical chain: `model_to_header` promotes the body model to
# `X-Model`, then `llmisvc_model_provider_resolver` strips the
# publisher prefix from the body only.
#
# Example request:
#
# curl -X POST http://localhost:8080/v1/chat/completions \
# -H "Content-Type: application/json" \
# -d '{"model":"publishers/rhoai/models/granite-3.1-8b","messages":[{"role":"user","content":"hi"}]}'
#
# Upstream body model becomes `granite-3.1-8b` while `X-Model`
# remains `publishers/rhoai/models/granite-3.1-8b`.

listeners:
- name: llmisvc-gateway
address: "0.0.0.0:8080" # dev value; binds all interfaces
filter_chains:
- rewrite-and-route

filter_chains:
- name: rewrite-and-route
filters:
- filter: model_to_header
header: X-Model
- filter: llmisvc_model_provider_resolver
header: X-Model
- filter: router
routes:
- path_prefix: "/"
cluster: provider
- filter: load_balancer
clusters:
- name: provider
endpoints:
- "127.0.0.1:3000"
Loading
Loading