feat: add model-selector RequestProcessor plugin#97
Draft
noyitz wants to merge 2 commits into
Draft
Conversation
Adds glue code to run the ModelSelector pipeline (Filter → Score → Pick) as a RequestProcessor plugin in the IPP pipeline. The plugin: - Implements framework.RequestProcessor - Takes candidate model names from plugin config (static list in this version; future versions may source from Datastore, CRD, or other dynamic sources) - Runs ModelSelector.Select() on each request - Writes selected model name to CycleState and request body - Uses a default max-score picker (replace with shared picker implementations once PR llm-d#74 merges) - Filters and scorers can be added via WithFilters() / WithScorers() once implementations are available Registered as "model-selector" plugin type in runner.go. Signed-off-by: Noy Itzikowitz <nitzikow@redhat.com>
github-actions
Bot
added
the
size/L
Collaborator
|
need to connect model selector as request plugin differently. |
Collaborator
|
need to take datastore from handle (PR #84). |
ModelSelector plugin now reads candidate models from the Datastore on each request instead of a static config list. - Constructor takes datastore.Datastore instead of []string - loadCandidateModels() reads all models from Datastore per request - Factory function is a placeholder until Handle exposes Datastore (PR llm-d#84) — TODO added for wiring via handle.Datastore() - Tests use datastore.NewStore() with pre-populated models Signed-off-by: Noy Itzikowitz <nitzikow@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds glue code to run the ModelSelector pipeline (Filter → Score → Pick) as a RequestProcessor plugin in the IPP pipeline. This allows model selection to be wired alongside other plugins like body-field-to-header in the request processing chain.
How it works:
ModelSelector.Select()runs the configured profile (Filter → Score → Pick)selected-modelkey) and the request bodymodelfieldWhat's included:
pkg/plugins/modelselector/plugin.go— RequestProcessor wrapper with factory functionpkg/plugins/modelselector/plugin_test.go— 7 testscmd/runner/runner.goExtension points:
WithFilters()/WithScorers()when implementations are ready.Why is this change needed?
This is the glue code needed to run the model-selector as a request plugin with one profile, as discussed with @shmuelk.
How was this tested?
7 tests covering:
Checklist
git commit -s) per DCOmake test)make lint)Related Issues
Related to the model selector work: #65, #73
Related to profiles design: #15, PR #92