Skip to content

feat: add LiteLLM as unified LLM provider#3182

Open
RheagalFire wants to merge 2 commits into
ModelEngine-Group:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as unified LLM provider#3182
RheagalFire wants to merge 2 commits into
ModelEngine-Group:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

@RheagalFire RheagalFire commented Jun 2, 2026

Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a model provider, giving nexent users access to 100+ LLM providers (OpenAI, Anthropic, Google Gemini, Azure, Bedrock, Ollama, etc.) through litellm.completion() as an SDK dependency.
  • Two integration points: backend provider for model discovery + SDK model for inference.

Changes

  • backend/consts/provider.py: added LITELLM = "litellm" to ProviderEnum
  • backend/services/providers/litellm_provider.py: new LiteLLMModelProvider for model discovery via /v1/models
  • backend/services/model_provider_service.py: wired LiteLLM into get_provider_models() dispatch
  • sdk/nexent/core/models/litellm_llm.py: new LiteLLMModel with streaming, token tracking, tool calling, drop_params=True
  • sdk/nexent/core/models/__init__.py: registered LiteLLMModel
  • test/sdk/test_litellm_model.py: 17 tests (init, streaming, credentials, edge cases, live E2E)

Tests

Unit tests (17/17 pass):

TestInit::test_basic PASSED
TestInit::test_with_credentials PASSED
TestCall::test_streaming_output PASSED
TestCall::test_api_key_forwarded PASSED
TestCall::test_api_key_omitted_when_none PASSED
TestCall::test_api_base_forwarded PASSED
TestCall::test_drop_params_set PASSED
TestCall::test_response_format PASSED
TestCall::test_stop_sequences PASSED
TestCall::test_token_tracking PASSED
TestEdgeCases::test_empty_stream PASSED
TestEdgeCases::test_chunk_without_choices PASSED
TestEdgeCases::test_context_length_exceeded PASSED
TestEdgeCases::test_auth_error_propagates PASSED
TestEdgeCases::test_import_error PASSED
TestEdgeCases::test_stop_event_interrupts PASSED
TestLiveE2E::test_live_streaming PASSED
============================== 17 passed in 3.64s ==============================

Live E2E (Anthropic claude-sonnet-4-6 via Azure Foundry, streaming):

Live E2E response: 'OK'

Risk / Compatibility

  • Additive only. Existing providers and OpenAIModel untouched.
  • litellm is lazy-imported so the base install is unaffected.
  • drop_params=True ensures cross-provider compatibility.

Example usage

from nexent.core.models import LiteLLMModel

# Any LiteLLM-supported provider
model = LiteLLMModel(
    model_id="anthropic/claude-sonnet-4-20250514",
    api_key="sk-ant-...",
)

# Or Google Gemini
model = LiteLLMModel(model_id="gemini/gemini-2.5-flash")
# export GEMINI_API_KEY=...

response = model([{"role": "user", "content": "Hello!"}])
print(response.content)

@RheagalFire

Copy link
Copy Markdown
Author

cc @Dallas98 @WMC001

headers["Authorization"] = f"Bearer {api_key}"

models_url = f"{base_url}/models"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

安全风险:verify=False 禁用了 SSL 证书验证,生产环境下容易遭受 MITM 攻击。项目已有 ssl_verify 配置字段,应该从 provider_config 读取并默认为 True


model_list = []
for item in data:
model_id = item.get("id", "")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

所有通过 LiteLLM 发现的模型都使用同一个 DEFAULT_LLM_MAX_TOKENS,不区分实际能力。gpt-4o 有 128K context,gpt-3.5-turbo 只有 16K。这会与 PR #3293 的 capacity management 系统冲突。建议从 LiteLLM 的 model cost map 查询实际能力,或允许 operator 覆盖。

"litellm is required for LiteLLMModel. "
"Install it with: pip install 'litellm>=1.80,<1.87'"
) from e

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ImportError__call__ 时才抛出,意味着错误只在第一次调用时才暴露。建议在 __init__ 时就验证 litellm 是否可用,实现更早的失败反馈。


return model_list

except Exception as e:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[代码规范] except Exception: 过于宽泛,建议捕获更具体的异常类型,避免掩盖潜在错误。

message.role = MessageRole.ASSISTANT
return message

except Exception as e:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[代码规范] except Exception: 过于宽泛,建议捕获更具体的异常类型,避免掩盖潜在错误。


await litellm.acompletion(**kwargs)
return True
except Exception as e:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[代码规范] except Exception: 过于宽泛,建议捕获更具体的异常类型,避免掩盖潜在错误。

@WMC001

WMC001 commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

LGTM. LiteLLM integration follows the existing provider pattern. No issues found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants