feat: add vLLM as a first-class Model Provider with dynamic model discovery#13910
Closed
pareek-ml wants to merge 4 commits into
Closed
feat: add vLLM as a first-class Model Provider with dynamic model discovery#13910pareek-ml wants to merge 4 commits into
pareek-ml wants to merge 4 commits into
Conversation
…covery Adds vLLM to Model Providers alongside Ollama, enabling centralized configuration (configure the server URL once, use everywhere) and dynamic model discovery via the OpenAI-compatible /v1/models API. Backend changes: - Register "vLLM" in LIVE_MODEL_PROVIDERS and MODEL_PROVIDER_METADATA with VLLM_API_BASE (required) and VLLM_API_KEY (optional) - Add fetch_live_vllm_models() for /v1/models discovery with auth header forwarding and /v1 deduplication - Register vLLM in the live-model dispatcher (get_live_models_for_provider) - Add vLLM single-fetch path in replace_with_live_models (no type distinction) - Add vLLM credential validation in validate_model_provider_key with distinct messages for auth errors, connection errors, and timeouts - Skip API key requirement for vLLM in get_llm and get_embeddings - Map "vLLM" to OpenAIEmbeddings in EMBEDDING_PROVIDER_CLASS_MAPPING - Add "vLLM Embeddings" to EMBEDDING_PARAM_MAPPINGS with base_url wiring - Add base_url resolution for vLLM Embeddings in get_embeddings - Ensure metadata-only providers (vLLM) appear in /api/v1/models response - Include MODEL_PROVIDER_METADATA keys in get_model_providers() Frontend changes: - Add vLLM icon mapping in use-get-model-providers.ts - Add VLLM_API_BASE to PROVIDER_VARIABLE_MAPPING - Skip model_type filtering for vLLM in modelInputComponent - Add isVllm rendering path in ModelSelection.tsx - Allow all model types for vLLM in ProviderList.tsx Tests: - 31 unit tests in test_vllm_provider.py covering metadata shape, live model fetching (both response formats, URL deduplication, auth header forwarding, error degradation), credential validation (happy path, 401/403, connection error, timeout, missing URL), and the API-key-optional contract Fixes: langflow-ai#12771 Based on: langflow-ai#12772
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughAdds vLLM as a first-class model provider. The backend registers vLLM in provider metadata, implements live model discovery from ChangesvLLM Provider Integration
Sequence Diagram(s)sequenceDiagram
participant UI as Frontend
participant API as list_models endpoint
participant Catalog as MODEL_PROVIDER_METADATA
participant vLLM as vLLM /v1/models
UI->>API: GET /models
API->>Catalog: get_model_provider_metadata()
Catalog-->>API: metadata including vLLM
API->>vLLM: GET /v1/models (fetch_live_vllm_models)
vLLM-->>API: model list
API-->>UI: filtered_models + vLLM entry (is_configured, is_enabled, models)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 5❌ Failed checks (1 warning, 4 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
enhancement
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/backend/base/langflow/api/v1/models.py`:
- Around line 203-223: The provider fallback in models.py is bypassing the
existing query-filter contract by appending metadata-only providers even when
model-level filters are active. Update the append logic in the loop that uses
get_model_provider_metadata(), existing_providers, and filtered_models so it
only adds providers when no model-specific filters are present, or re-run the
same filtering criteria from get_unified_models_detailed(...) before appending.
Make sure provider-only fallback still works for empty static-model providers
like vLLM, but /models requests with model_name or metadata filters do not
return empty providers that cannot satisfy the query.
In `@src/frontend/src/constants/providerConstants.ts`:
- Line 33: The vLLM entry in PROVIDER_VARIABLE_MAPPING is incorrectly forcing
the UI to treat VLLM_API_BASE as the synthetic “API Key” fallback in
useProviderConfiguration. Remove this static mapping for vLLM, and let the
provider configuration come from the backend payload instead so the UI can honor
VLLM_API_BASE plus optional VLLM_API_KEY metadata. If vLLM needs special
handling, add an explicit multi-field mapping rather than routing it through the
single secret-field fallback.
In `@src/frontend/src/modals/modelProviderModal/components/ModelSelection.tsx`:
- Line 292: The vLLM branch in ModelSelection is bypassing the same availability
bookkeeping used for metadata.model_type, so availableModels can render while
noModelsAvailable and search visibility still act as if nothing is available.
Update the logic around isVllm, availableModels, and the search/empty-state
checks so the vLLM path treats availableModels.length > 0 as the source of truth
for availability and query matching before branching, keeping the rendered list,
search box, and empty state consistent.
In `@src/lfx/src/lfx/base/models/model_utils.py`:
- Around line 714-722: The models listing request in model_utils uses
VLLM_API_BASE directly in requests.get, which allows user-scoped URLs to probe
arbitrary hosts. Update the model discovery flow around the models_url
construction and requests.get call to use the same SSRF-safe URL
validation/normalization used by the Ollama and LMStudio paths, or add
equivalent allowlist/private-range blocking before any network request is made.
Keep the fix localized to the model utility helper so the base URL is rejected
or sanitized before headers and the GET request are built.
In `@src/lfx/src/lfx/base/models/unified_models/instantiation.py`:
- Around line 476-486: The vLLM Embeddings `api_base` fallback in
`instantiation.py` is looking up provider-specific variables that do not exist
for this provider, so update the `base_url_value` resolution in the `vLLM
Embeddings` branch to use the same source as the main vLLM provider or add a
matching metadata key. Specifically, adjust the `provider_vars =
unified_models_module.get_all_variables_for_provider(...)` and
`_env_if_allowed(...)` lookups so they reference the correct provider symbol and
environment variable names, and keep the assignment into
`kwargs[param_mapping["api_base"]]` unchanged once the right value is resolved.
In `@src/lfx/tests/unit/base/models/test_vllm_provider.py`:
- Around line 14-19: The new test module has Ruff issues that need cleanup: fix
the import ordering/formatting in the module import block, adjust the docstrings
in the test cases to satisfy Ruff’s docstring style rules, and rename the
intentionally unused parameters in the fake_get callbacks to use the
conventional unused prefix. Use the test module’s top-level imports, the
affected test docstrings, and the fake_get helper callbacks as the places to
update so the file becomes lint-clean.
- Around line 496-526: The current test only covers get_llm, so add a companion
contract test for get_embeddings in test_vllm_provider.py to lock down the vLLM
API-key-optional and base-url behavior. Mirror the existing setup pattern by
patching the same unified_models_module helpers and calling get_embeddings with
a vLLM model_selection entry, then assert the embeddings client is built with
the expected arguments and does not require an API key. Use the get_embeddings
symbol alongside get_llm so the test suite covers both changed code paths rather
than only the text-model path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 86f6ad60-2030-4e7f-adbe-c450bafd5efc
📒 Files selected for processing (13)
src/backend/base/langflow/api/v1/models.pysrc/frontend/src/components/core/parameterRenderComponent/components/modelInputComponent/index.tsxsrc/frontend/src/constants/providerConstants.tssrc/frontend/src/controllers/API/queries/models/use-get-model-providers.tssrc/frontend/src/modals/modelProviderModal/components/ModelSelection.tsxsrc/frontend/src/modals/modelProviderModal/components/ProviderList.tsxsrc/lfx/src/lfx/base/models/model_metadata.pysrc/lfx/src/lfx/base/models/model_utils.pysrc/lfx/src/lfx/base/models/unified_models/class_registry.pysrc/lfx/src/lfx/base/models/unified_models/credentials.pysrc/lfx/src/lfx/base/models/unified_models/instantiation.pysrc/lfx/src/lfx/base/models/unified_models/provider_queries.pysrc/lfx/tests/unit/base/models/test_vllm_provider.py
- Gate metadata-only provider injection in /api/v1/models on absence of model-level filters (model_name, tool_calling, etc.) so filtered requests don't return empty vLLM entries that cannot satisfy the query - Remove vLLM from PROVIDER_VARIABLE_MAPPING; let the backend-provided multi-field metadata (VLLM_API_BASE + optional VLLM_API_KEY) drive the UI instead of the single-secret-field fallback - Fix ModelSelection.tsx vLLM availability bookkeeping: use availableModels.length for noModelsAvailable and apply search query filtering via vllmFilteredModels so the search box and empty state are consistent - Add SSRF protection to fetch_live_vllm_models via validate_url_for_ssrf before the requests.get call (respects LANGFLOW_SSRF_PROTECTION_ENABLED and LANGFLOW_SSRF_ALLOWED_HOSTS) - Fix vLLM Embeddings base_url resolution in instantiation.py to look up provider variables under 'vLLM' (not 'vLLM Embeddings') and use VLLM_API_BASE (not the undefined VLLM_EMBEDDINGS_API_BASE) - Fix all Ruff violations in test_vllm_provider.py (I001 import sort, D205/D209/D403 docstring style, ARG001 unused args prefixed with _) - Add get_embeddings companion test covering the API-key-optional and base_url resolution contracts for vLLM Embeddings (32 tests total)
pareek-ml
force-pushed
the
feat/vllm-model-provider
branch
from
622ecef to
e7e495c
Compare
github-actions
Bot
added
enhancement
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
Author
|
Hey @ogabrielluiz @erichare @Cristhianzl — this PR adds vLLM as a first-class Model Provider (dynamic model discovery, embeddings, credential validation). All 111 CI checks pass and CodeRabbit has no open issues. Would appreciate a review when you get a chance! |
This was referenced Jun 30, 2026
erichare
added a commit
that referenced
this pull request
Jun 30, 2026
…13910 (#13919) feat(bundles): add vLLM model-provider bundle (lfx-vllm) Ships vLLM as a standalone Langflow Extension Bundle built on the provider-registry seam: its extension.json declares a providers[] entry that registers a vLLM model provider into the unified model system, with zero core edits. vLLM is OpenAI-compatible, so the provider reuses ChatOpenAI / OpenAIEmbeddings and discovers served models live from /v1/models (SSRF-guarded); the same model is offered for both Language Model and Embedding Model use, and no API key is required for local servers. Inherits the original vLLM provider from #13910 by Yash Pareek (@pareek-ml), reconstructed as a bundle so it no longer edits the 12 core files that PR did. Co-authored-by: Yash Pareek <yash.pareek@usi.ch>
Author
|
Superseded by #13919 , which ships vLLM as a standalone bundle without core edits. Closing in favor of it. |
erichare
added a commit
that referenced
this pull request
Jun 30, 2026
* feat(lfx): let extension bundles register model providers Model providers were hardcoded across lfx core (MODEL_PROVIDER_METADATA, the class-import registries, LIVE_MODEL_PROVIDERS, and the credential / instantiation / live-discovery dispatch), so a third party could not add a provider without editing core -- adding vLLM (#13910) touched 12 files. Introduce a supported extension point: an extension manifest may declare a `providers[]` block, which the loader merges into the core provider tables in place via a new `provider_registry`. Every existing accessor (the lru_cached variable maps, the shared `model_provider_metadata` reference, the /api/v1/models surface) then sees a registered provider with no further edits. - new lfx/base/models/provider_registry.py: ProviderSpec + register_provider with core-wins precedence, failure isolation, and a clear() test seam; live-discovery and validator callables resolved lazily by dotted path. - manifest: declarative providers[]; allow a provider-only extension (empty bundles); require at least one bundle or provider. - loader: register manifest providers during load_extension (the startup chokepoint), failure-isolated as warnings. - dispatch seams made registry-aware: get_model_providers union, live-model discovery fallback, credential-validation fallback, api-key-optional check. - zero added cost and byte-identical behavior when no provider bundle is installed. Groundwork for shipping providers as standalone bundles (e.g. lfx-vllm, superseding #13910). * feat(lfx): apply registered providers' connection variables generically get_llm/get_embeddings resolved base_url (and attribution headers) only via hardcoded per-provider branches, so a bundle-contributed OpenAI-compatible provider could not point its client at a custom endpoint -- it would silently hit the default API. Add a generic, is_registered-gated step that applies each non-secret metadata variable to its declared langchain_param (base_url localhost-rewritten) or HTTP header, mirroring the core branches. Core providers keep their explicit branches, so behavior is unchanged. Completes the provider-bundle seam so an OpenAI-compatible provider (e.g. vLLM) works fully from a bundle with zero core edits. * feat(lfx): pass a placeholder api key for key-optional registered providers langchain_openai's ChatOpenAI / OpenAIEmbeddings raise when constructed with api_key=None, so an OpenAI-compatible provider that declared api_key_required=False (e.g. a local vLLM server without auth) would fail to instantiate even though the model system correctly skips the "API key required" error. Pass a non-empty "EMPTY" placeholder for such providers so the client constructs. Only applies to registered (bundle) providers that opted out of API keys; core providers are unaffected. * feat(lfx): apply the api key for key-optional registered embedding providers get_embeddings only passed an API key when the provider's param_mapping declared an explicit "api_key" slot. Bundle providers can omit that slot, so apply the resolved key (or the api-key-optional placeholder) under the OpenAI-compatible "api_key" kwarg for registered providers -- mirroring how get_llm already resolves the key. Lets an OpenAI-compatible embedding provider (e.g. vLLM) work from a bundle without that slot. * feat(bundles): add vLLM model-provider bundle (lfx-vllm) — inherits #13910 (#13919) feat(bundles): add vLLM model-provider bundle (lfx-vllm) Ships vLLM as a standalone Langflow Extension Bundle built on the provider-registry seam: its extension.json declares a providers[] entry that registers a vLLM model provider into the unified model system, with zero core edits. vLLM is OpenAI-compatible, so the provider reuses ChatOpenAI / OpenAIEmbeddings and discovers served models live from /v1/models (SSRF-guarded); the same model is offered for both Language Model and Embedding Model use, and no API key is required for local servers. Inherits the original vLLM provider from #13910 by Yash Pareek (@pareek-ml), reconstructed as a bundle so it no longer edits the 12 core files that PR did. Co-authored-by: Yash Pareek <yash.pareek@usi.ch> * fix(lfx): address CodeRabbit review on the provider seam - model_utils: normalize non-list live-discovery returns to [] (list[dict] contract) - provider_registry: validate lazy-import/embedding keys before mutating any global table (reject unknown or conflicting model/embedding classes and embedding_param_key collisions, so clear() can't drop a core mapping); _resolve_callable verifies the target is callable; callable resolution now catches broad import-time failures (SyntaxError, side effects) and degrades to None - credentials: normalize non-ValueError bundle-validator failures to ValueError so they can't escape as an unhandled 500 - loader: surface skipped (name-collision) provider registrations as a typed provider-skipped warning instead of returning silent success - errors: register provider-invalid / provider-skipped codes + format templates - tests: cover the register-time warning-isolation and collision paths, unknown/conflicting keys, non-callable and non-list discovery, and validator normalization * docs(bundle-api): changelog for the providers[] manifest surface Manifest providers[] (model-provider registration) and the optional 0-or-1 bundles[] (provider-only extensions) touch in-scope BUNDLE_API files (manifest.py, _orchestrator.py, errors.py); record the additive contract change + the new provider-invalid / provider-skipped codes so the changelog gate passes. No BUNDLE_API_VERSION bump (all additive). * fix(lfx): correct provider api-key resolution + provider-only discovery Addresses two review findings on the provider seam: - [P1] get_model_provider_variable_mapping() fell back to the FIRST provider variable when no *required* secret existed, so a provider whose API key is an optional secret (e.g. vLLM's VLLM_API_KEY) mapped to its required non-secret base URL. get_api_key_for_provider then resolved and forwarded the endpoint as the bearer token, skipping the "EMPTY" placeholder. Now prefer a required secret, then any secret, then the first variable. Covered by a test that exercises the real resolver (no patched stub). - [P2] discover_installed_extensions / discover_seed_extensions indexed manifest.bundles[0], crashing with IndexError on a valid provider-only manifest (bundles=[]) and breaking `lfx extension list` / registry discovery. Guard the empty-bundle case; DiscoveredExtension.bundle_name and registry.Extension.bundle_name are now str | None, and the list CLI renders a dash for provider-only extensions. BUNDLE_API.md changelog updated for the discovery/registry surface nullability. --------- Co-authored-by: Yash Pareek <yash.pareek@usi.ch>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds vLLM as a first-class Model Provider in Langflow — configure your vLLM server URL once in Settings → Model Providers → vLLM, and all components automatically pick it up, just like Ollama.
Fixes #12771
Why vLLM?
vLLM is one of the most popular open-source LLM serving frameworks for self-hosted inference. Langflow already ships
vLLMandvLLM Embeddingscomponents and a registered vLLM icon, but vLLM was absent from Model Providers — users had to manually enter the API base URL in every component instance.This PR is a clean rebase of #12772 on current
main, with all CodeRabbit issues resolved and full test coverage added.What This Enables
VLLM_API_BASEonce, all components pick it up/v1/models(no static catalog needed)Authorization: BearerautomaticallyChanges
src/lfx/.../model_metadata.py"vLLM"toLIVE_MODEL_PROVIDERSandMODEL_PROVIDER_METADATAsrc/lfx/.../model_utils.pyfetch_live_vllm_models()— discovery via/v1/models, handles both OpenAI dict and plain list formats, API key auth,/v1deduplicationsrc/lfx/.../credentials.pysrc/lfx/.../class_registry.py"vLLM"→OpenAIEmbeddings; add"vLLM Embeddings"param mappingsrc/lfx/.../instantiation.pysrc/lfx/.../provider_queries.pyMODEL_PROVIDER_METADATAproviders inget_model_providers()src/backend/.../models.pysrc/frontend/.../use-get-model-providers.tssrc/frontend/.../providerConstants.tsvLLM: "VLLM_API_BASE"src/frontend/.../modelInputComponent/index.tsxsrc/frontend/.../ModelSelection.tsxsrc/frontend/.../ProviderList.tsxsrc/lfx/tests/.../test_vllm_provider.pyDesign Decisions
mapping_field: "vllm_base_url"(contains"base_url") — recognized byget_provider_param_mapping's URL classifier, sobase_url_paramis correctly wired to downstream consumers inmodel_catalog.pyandflow_preparation.pymodel_param: "model"— uses the modernlangchain_openai.ChatOpenAIparameter (not the deprecatedmodel_name)/v1/modelsendpoint — correct OpenAI-compatible path; handles the common case where users providehttp://host:8000/v1by checking for the suffix before appendingreplace_with_live_modelsfetches once to avoid duplicatesChatOpenAI(LLM) andOpenAIEmbeddings(embeddings)Test Coverage
Covers:
LIVE_MODEL_PROVIDERS,MODEL_PROVIDER_METADATA,EMBEDDING_PROVIDER_CLASS_MAPPING)mapping_fieldcontains"base_url"so the param classifier worksget_model_providers()includes vLLM despite no static catalogfetch_live_vllm_models: OpenAI dict format, plain list format, alphabetical sort,/v1deduplication, auth header forwarding, no-auth path, connection error, timeout, bad payload degradation, LLM/embeddings same outputvalidate_model_provider_key: happy path, correct/v1/modelsendpoint, API key forwarding, no-auth path, 401, 403, connection error, timeout,/v1deduplicationget_llmdoes not raise when no API key is set for vLLMRelationship to #12772
This is a clean reimplementation on current
mainresolving all issues that prevented #12772 from merging:mapping_field,model_param, endpoint path, API key forwarding)Summary by CodeRabbit
New Features
Bug Fixes
Tests