Add a proposal for generic model routing support. - #643
Conversation
Model selection, for Mixture of Models, may incorporate many approaches--complexity analysis, semantic routing, etc...--from many sources. Meanwhile model routing hygeine and policy enforcement is needed across all possible approaches. This proposal aims to define an API contract, allowing model selection approaches to plug and play with Praxis's Mixture of Models support. That is, a model or list of models is proposed and the work of mutating a request or selecting a corresponding cluster to route to is handled downstream. Signed-off-by: usize <mofoster@redhat.com>
|
This proposal doesn't include a "How" section, so there's no shape to the API contract that's suggested. Here are some sketches of what sorts of policies we'd want to support, that might help guide the way: https://gist.github.com/usize/6925f78eca5b2291caac651c7ccf8eb6 |
praxis-bot
left a comment
praxis-bot
left a comment
There was a problem hiding this comment.
Review Summary
Strong proposal with well-researched prior art and a clear motivation. The separation between model selection and model routing is well-grounded in established proxy-native patterns (GIE, Envoy AI Gateway), and the policy gate/library framing draws effective parallels from Kubernetes admission control, OPA, and the scheduler's constraint-vs-preference split.
One design gap: the API contract is described in terms of a single model suggestion, but the proposal does not address multi-selector conflict (multiple selector filters in the same pipeline). This is a contract-level question that belongs in the What? section.
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 1 |
| Medium | 3 |
|
|
||
| ## What? | ||
|
|
||
| Separate model selection from and the mechanisms necessary to route to a model in the |
There was a problem hiding this comment.
[Medium] "Separate model selection from and the mechanisms" reads as if a word was dropped during editing. Suggest:
Separate model selection from the mechanisms necessary to route to a model in the filter pipeline.
| filter pipeline. These mechanisms include alias resolution, fact promotion to | ||
| headers and/or metadata, safety validation, and policy enforcement. | ||
|
|
||
| Today, any filter that wants to route a request to a different model must reimplement that machinery (or skip it). This proposal introduces a shared rewrite mechanism that any upstream filter can invoke by writing a model suggestion to filter |
There was a problem hiding this comment.
[Large] The contract is described in terms of "a model suggestion" (singular) written to filter metadata by "any upstream filter." When multiple selector filters run in the same pipeline and each writes a suggestion, what semantics apply? Last-writer-wins? Priority ordering? List-of-candidates?
This is a contract-level question, not an implementation detail -- selector authors need to know whether their suggestion can be overwritten by a later filter before the How? section lands. Please add a sentence here (or in Goals/Non-Goals) clarifying the expected cardinality and conflict semantics, even if the answer is simply "last suggestion wins."
|
|
||
| - Classification and selection strategy. Model selection is treated as a black box so | ||
| operators can integrate whichever approach fits their needs. | ||
| - Cross-request state. Managing state across the turns of a conversation (routing momentum, context dilution, session coherence) is out of scope. Selectors that need cross-request memory build on the state primitives from proposals #412 (storage layer) and #432 (request extensions). |
There was a problem hiding this comment.
[Medium] This line runs to ~287 characters. Several other lines in the What? and Motivation sections also exceed the 80-character Markdown line width convention (lines 23, 27, 42, 61--62). The Prior Art and Policy sections are correctly wrapped at ~50 characters -- please wrap the remaining long lines to match.
Also, line 43 has trailing whitespace after the period.
| metadata promotion, safety validation, policy | ||
| checks). | ||
| 2. Bypass the model field entirely and select a | ||
| cluster directly--conflating model identity |
There was a problem hiding this comment.
[Medium] Items 1 and 3 in this list end with periods; this item does not. Add a trailing period after "topology" for consistency.
praxis-bot
left a comment
praxis-bot
left a comment
There was a problem hiding this comment.
The prior art taxonomy and the gate-plus-library argument are well-constructed. Two new findings beyond the previous review.
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 1 |
| Medium | 1 |
Large: The proposal defines a policy gate that can reject model suggestions but does not specify the rejection semantics. Whether rejection is terminal or recoverable determines whether the policy library is mandatory or optional for reliable selectors -- a contract-level distinction that should be settled in the What? section before How? work begins.
Medium: The GIE prior art description names a component ("InferenceModelRewrite") that does not match any GIE CRD or component. The actual CRD is InferenceModel.
| correctness -- the gate guarantees that -- but it | ||
| should consult the policy for quality, since | ||
| rejected suggestions waste requests. That is why | ||
| this proposal treats the gate and the library as |
There was a problem hiding this comment.
[Large] "Rejected suggestions waste requests" implies the gate can reject, but the proposal does not define what happens when it does. Is rejection terminal (request fails with an error to the client), does the request fall through with the original client-specified model, or is there a retry/feedback loop?
This is a contract-level question: if rejection is terminal, the library becomes mandatory for reliable selectors because a rejected suggestion means a failed request. If rejection triggers a fallback (e.g., pass-through with the original model), the library is an optimization and selectors are best-effort. Define the gate's rejection behavior in the Goals or What? section so that the How? design can build on clear semantics.
| metrics (communicating its choice via the | ||
| `x-gateway-destination-endpoint` header), and | ||
| InferenceModelRewrite that handles model name | ||
| rewriting in the body as a [separate CRD][gie-rewrite]. |
There was a problem hiding this comment.
[Medium] "InferenceModelRewrite" does not appear as a component or CRD name in the Gateway API Inference Extension. The CRD that defines model-to-model mapping is InferenceModel, which uses a targetModel field for name translation. The linked adapter-rollout guide discusses LoRA adapter deployment using InferenceModel, not a standalone rewrite CRD. Correct the component name to InferenceModel and update the reference link to point to the InferenceModel API type documentation.
2 participants
Summary
Model selection, for Mixture of Models, may incorporate many approaches--complexity analysis, semantic routing, etc...--from many sources. Meanwhile model routing hygeine and policy enforcement is needed across all possible approaches.
This proposal aims to define an API contract, allowing model selection approaches to plug and play with Praxis's Mixture of Models support.
That is, a model or list of models is proposed and the work of mutating a request or selecting a corresponding cluster to route to is handled downstream.
Related issue
#74