feat: add LiteLLM as unified LLM provider#3182
Conversation
| headers["Authorization"] = f"Bearer {api_key}" | ||
|
|
||
| models_url = f"{base_url}/models" | ||
|
|
There was a problem hiding this comment.
安全风险:verify=False 禁用了 SSL 证书验证,生产环境下容易遭受 MITM 攻击。项目已有 ssl_verify 配置字段,应该从 provider_config 读取并默认为 True。
|
|
||
| model_list = [] | ||
| for item in data: | ||
| model_id = item.get("id", "") |
There was a problem hiding this comment.
所有通过 LiteLLM 发现的模型都使用同一个 DEFAULT_LLM_MAX_TOKENS,不区分实际能力。gpt-4o 有 128K context,gpt-3.5-turbo 只有 16K。这会与 PR #3293 的 capacity management 系统冲突。建议从 LiteLLM 的 model cost map 查询实际能力,或允许 operator 覆盖。
| "litellm is required for LiteLLMModel. " | ||
| "Install it with: pip install 'litellm>=1.80,<1.87'" | ||
| ) from e | ||
|
|
There was a problem hiding this comment.
ImportError 在 __call__ 时才抛出,意味着错误只在第一次调用时才暴露。建议在 __init__ 时就验证 litellm 是否可用,实现更早的失败反馈。
|
|
||
| return model_list | ||
|
|
||
| except Exception as e: |
There was a problem hiding this comment.
[代码规范] except Exception: 过于宽泛,建议捕获更具体的异常类型,避免掩盖潜在错误。
| message.role = MessageRole.ASSISTANT | ||
| return message | ||
|
|
||
| except Exception as e: |
There was a problem hiding this comment.
[代码规范] except Exception: 过于宽泛,建议捕获更具体的异常类型,避免掩盖潜在错误。
|
|
||
| await litellm.acompletion(**kwargs) | ||
| return True | ||
| except Exception as e: |
There was a problem hiding this comment.
[代码规范] except Exception: 过于宽泛,建议捕获更具体的异常类型,避免掩盖潜在错误。
|
LGTM. LiteLLM integration follows the existing provider pattern. No issues found. |
Summary
litellm.completion()as an SDK dependency.Changes
backend/consts/provider.py: addedLITELLM = "litellm"toProviderEnumbackend/services/providers/litellm_provider.py: newLiteLLMModelProviderfor model discovery via/v1/modelsbackend/services/model_provider_service.py: wired LiteLLM intoget_provider_models()dispatchsdk/nexent/core/models/litellm_llm.py: newLiteLLMModelwith streaming, token tracking, tool calling,drop_params=Truesdk/nexent/core/models/__init__.py: registeredLiteLLMModeltest/sdk/test_litellm_model.py: 17 tests (init, streaming, credentials, edge cases, live E2E)Tests
Unit tests (17/17 pass):
Live E2E (Anthropic claude-sonnet-4-6 via Azure Foundry, streaming):
Risk / Compatibility
OpenAIModeluntouched.litellmis lazy-imported so the base install is unaffected.drop_params=Trueensures cross-provider compatibility.Example usage