From 0086ec8810e403865dc5b5770eb918ce79d44357 Mon Sep 17 00:00:00 2001 From: usize Date: Sun, 2 Aug 2026 14:53:46 -0700 Subject: [PATCH] Add a proposal for generic model routing support. Model selection, for Mixture of Models, may incorporate many approaches--complexity analysis, semantic routing, etc...--from many sources. Meanwhile model routing hygeine and policy enforcement is needed across all possible approaches. This proposal aims to define an API contract, allowing model selection approaches to plug and play with Praxis's Mixture of Models support. That is, a model or list of models is proposed and the work of mutating a request or selecting a corresponding cluster to route to is handled downstream. Signed-off-by: usize --- .../00074_model-rewrite-abstraction.md | 275 ++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 docs/proposals/00074_model-rewrite-abstraction.md diff --git a/docs/proposals/00074_model-rewrite-abstraction.md b/docs/proposals/00074_model-rewrite-abstraction.md new file mode 100644 index 0000000000..f94fad43f5 --- /dev/null +++ b/docs/proposals/00074_model-rewrite-abstraction.md @@ -0,0 +1,275 @@ +--- +issue: https://github.com/praxis-proxy/ai/issues/74 +discussion: https://github.com/praxis-proxy/praxis/discussions/838 +status: proposed +authors: + - usize +graduation_criteria: + - What? and Why? accepted by stakeholders + - How? section with requirements and design +stakeholders: + - shaneutt + - leseb + - bentito + - cnuland + - caldeirav + - franciscojavierarceo +--- + +# Model Rewrite Abstraction + +## What? + +Separate model selection from and the mechanisms necessary to route to a model in the +filter pipeline. These mechanisms include alias resolution, fact promotion to +headers and/or metadata, safety validation, and policy enforcement. + +Today, any filter that wants to route a request to a different model must reimplement that machinery (or skip it). This proposal introduces a shared rewrite mechanism that any upstream filter can invoke by writing a model suggestion to filter +metadata, leaving the mutation and policy enforcement to dedicated downstream machinery. + +### Goals + +- Define an API contract between model-selecting + filters and model-routing machinery. +- Provide a final policy gate before model rewriting + so that misconfigured or buggy selectors cannot + route requests to disallowed models. +- Expose policy configuration as a library to filter authors so + selectors can make informed suggestions. + +### Non-Goals + +- Classification and selection strategy. Model selection is treated as a black box so + operators can integrate whichever approach fits their needs. +- Cross-request state. Managing state across the turns of a conversation (routing momentum, context dilution, session coherence) is out of scope. Selectors that need cross-request memory build on the state primitives from proposals #412 (storage layer) and #432 (request extensions). + +## Why? + +### Motivation + +Mixture-of-models routing is a core goal of the AI +gateway (issue #74). Multiple filters will need to +select models: semantic routers, complexity +classifiers, cost-aware routers, latency-aware +routers, LoRA adapter routers, and more. Each of +these filters solves a different selection problem, +but all share the same downstream need: rewrite the +`model` field in the request body and enforce +organizational policy on which models are permitted. + +There is no policy gate anywhere in Praxis today that can answer "is this +model allowed for this request?" PR #446 demonstrated the problem concretely. That +PR implemented a semantic router filter that could not enforce any model access policies. It circumvented the difficulties of model re-writing via cluster selection--introducing an assumption that a particular cluster represents a particular model. + +Without a shared rewrite mechanism, every new +model-selecting filter must either: + +1. Reimplement the selection-facing half of the + rewrite pipeline (alias resolution, header and + metadata promotion, safety validation, policy + checks). +2. Bypass the model field entirely and select a + cluster directly--conflating model identity + with backend topology +3. Depend on the existing `ModelRewriteFilter`, + which is Responses-API-specific and gates on `POST /v1/responses`. + +None of these options scale to the routing +strategies enumerated in #74. + +#### Why not Guardrails? + +Guardrails filters can not close the +policy gap. Guardrails match request content +against configured patterns; they cannot evaluate +"is this model allowed for this request" at the +moment a model is chosen, and they cannot feed +policy back to a selector so it avoids suggesting +models that will be rejected. The gate belongs at +the rewrite decision point, and the policy +configuration must be consumable as a library by +upstream selectors. + +### Prior Art + +AI gateways and routing projects fall into three +architectural categories on this question. Where +the boundary lands depends on whether the system +mutates in-flight requests (as Praxis does) or +constructs fresh outbound requests. + +**SDK-style gateways ([LiteLLM][litellm-arch], +[Portkey][portkey-plugins], [MLflow][mlflow-gw])** +do not separate selection from rewriting. These +projects construct fresh outbound HTTP requests +for each provider call, so there is no in-flight +body to mutate. + +Portkey is a partial exception: its +`beforeRequestHooks` mechanism runs after target +selection but before the provider call, allowing +guardrail hooks to deny or transform requests. +This is the closest any SDK-style gateway comes +to an intervention point between selection and +execution. + +**Recommendation-only projects ([Not Diamond][notdiamond], +[RouteLLM][routellm], [Aurelio Semantic Router][semantic-router])** +achieve total separation by scope limitation -- they +only select, never rewrite. Not Diamond returns a +model recommendation; the caller handles everything +else. RouteLLM is a binary strong/weak classifier +that delegates to LiteLLM for the actual API call. +Aurelio Semantic Router classifies intent via +embedding similarity and returns a route name. None +of these projects touch request bodies. + +**Proxy-native projects ([Gateway API Inference +Extension][gie], [Envoy AI Gateway][envoy-ai-gw])** +are the direct analogues to Praxis, and both +implement explicit separation between selection +and rewriting: + +- The Gateway API Inference Extension (GIE) + decomposes the problem into four distinct + components: a Body-Based Router that extracts + the model name and promotes it to a header (pure + extraction, no mutation), HTTPRoute rules that + route based on that header, an Endpoint Picker + that selects a specific pod based on runtime + metrics (communicating its choice via the + `x-gateway-destination-endpoint` header), and + InferenceModelRewrite that handles model name + rewriting in the body as a [separate CRD][gie-rewrite]. + +- The Envoy AI Gateway uses [two ext_proc + phases][envoy-ai-gw-dataplane]: a router-level + phase that extracts the model name and sets + `x-ai-eg-model` for routing, and an + upstream-level phase that handles schema + translation, credential injection, and [model + name override][envoy-ai-gw-vnm] via the + `modelNameOverride` field on `AIGatewayRoute` + backend refs. + +The suggest-then-apply pattern -- where one +component suggests a model or endpoint via +metadata and a separate component applies the +mutation -- is the established architecture in the +proxy-native space. It exists because proxies that +mutate in-flight requests face real per-site +engineering costs (JSON parse, re-serialize, +content-length fixup, header promotion) that +SDK-style gateways avoid by constructing fresh +requests. + +### Policy: Gate and Library + +A recurring concern with separating selection from +rewriting is that the selector seemingly *needs* +to know the policy: if requests flagged with PII +must not reach external models, doesn't the thing +picking the model need to know that? Established +policy systems answer with a consistent split: +enforce authoritatively at a single gate, and +expose the same policy to callers as a library or +discovery API so they can make informed choices. +Correctness never depends on the caller's +cooperation; caller awareness exists to avoid +wasted work. + +- **[Kubernetes admission control][k8s-admission]** + runs mutating webhooks first and validating + webhooks only after all mutation completes, so + policy always evaluates the final object no + matter which component mutated it. Clients can + pre-flight a check through the + `SelfSubjectAccessReview` API + ([`kubectl auth can-i`][k8s-authz]); + enforcement at the API server is unaffected by + whether clients bother. +- **[The Kubernetes scheduler][k8s-scheduler]** + separates hard constraints from preferences: + filter plugins eliminate nodes that cannot run a + pod, then score plugins rank the survivors. Any + scorer composes with any filter set because + constraint satisfaction is not the scorer's job. + Under this proposal, model selection is a + preference; model policy is a constraint. +- **Open Policy Agent** runs the same policy + document in both positions: [embedded as a + library][opa-integration] so callers evaluate + decisions locally (including [partial + evaluation][opa-partial-eval] to derive the set + of permitted options up front), and deployed as + a boundary enforcement point ([Envoy + `ext_authz`][opa-envoy], Kubernetes admission). + Selector and gate cannot drift apart because + they share one source of truth. +- **LiteLLM's proxy** enforces key- and + team-level [model access groups][litellm-access] + at call time and exposes the same policy through + model discovery (`/models`), so a caller can + enumerate what it may use before choosing. + +The selector does not need to know the policy for +correctness -- the gate guarantees that -- but it +should consult the policy for quality, since +rejected suggestions waste requests. That is why +this proposal treats the gate and the library as +one deliverable rather than trusting selectors to +get policy right on their own. + +### User Stories + +- As a filter author building a semantic router, I + want to suggest a model for a request without + reimplementing alias resolution, header and + metadata promotion, and policy checks so that I + can focus on classification logic. +- As a platform engineer, I want a single policy + gate that enforces which models are permitted for + a request regardless of which upstream filter + selected the model, so that organizational access + controls cannot be bypassed by a misconfigured + classifier. +- As a proxy operator deploying multiple + model-selecting strategies (semantic routing, + cost-aware routing, complexity classification), I + want these strategies to compose with a single + rewrite mechanism so that I do not need to + configure body mutation independently for each + one. +- As a filter author, I want access to the policy + configuration (which models are allowed for which + users, groups, or request properties) as a + library so that my selector can make informed + suggestions rather than suggesting models that + will be rejected downstream. + +## How? + +> **Note:** do not include this section in the first PR. +> Submit What? and Why? first. Add How? in a follow-up +> PR after the proposal direction is accepted. + + + +[litellm-arch]: https://docs.litellm.ai/docs/proxy/architecture +[portkey-plugins]: https://github.com/Portkey-AI/gateway/blob/main/plugins/README.md +[mlflow-gw]: https://mlflow.org/docs/latest/genai/governance/ai-gateway/ +[notdiamond]: https://docs.notdiamond.ai/docs/quickstart-routing +[routellm]: https://github.com/lm-sys/RouteLLM +[semantic-router]: https://github.com/aurelio-labs/semantic-router +[gie]: https://gateway-api-inference-extension.sigs.k8s.io/ +[gie-rewrite]: https://gateway-api-inference-extension.sigs.k8s.io/guides/adapter-rollout/ +[envoy-ai-gw]: https://aigateway.envoyproxy.io/docs/concepts/architecture/ +[envoy-ai-gw-dataplane]: https://aigateway.envoyproxy.io/docs/concepts/architecture/data-plane/ +[envoy-ai-gw-vnm]: https://aigateway.envoyproxy.io/docs/capabilities/traffic/model-name-virtualization/ +[k8s-admission]: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ +[k8s-authz]: https://kubernetes.io/docs/reference/access-authn-authz/authorization/ +[k8s-scheduler]: https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/ +[opa-integration]: https://www.openpolicyagent.org/docs/integration +[opa-partial-eval]: https://www.openpolicyagent.org/docs/filtering/partial-evaluation +[opa-envoy]: https://www.openpolicyagent.org/docs/envoy +[litellm-access]: https://docs.litellm.ai/docs/proxy/model_access