Add universal OpenAI-compatible gateway support#43
Open
bigtcze wants to merge 4 commits into
Open
Conversation
Allow routing all API requests through a LiteLLM proxy server, enabling use of any LLM provider (Anthropic, local models, etc.) via a single OpenAI-compatible endpoint. Supports both Chat Completions and Responses API through the proxy. LiteLLM is mutually exclusive with Infomaniak but works alongside the Responses API toggle.
Add LiteLLM Proxy Support
|
Important Review skippedToo many files! This PR contains 171 files, which is 21 over the limit of 150. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (171)
You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds LiteLLM proxy support: DB migration and model fillable fields, controller endpoints and generate-flow branching for LiteLLM (responses & chat), a POST route for model discovery, and UI controls + JS to configure and fetch LiteLLM models. Changes
Sequence DiagramsequenceDiagram
participant Frontend
participant Controller
participant Cache
participant LiteLLM
rect rgba(200, 150, 100, 0.5)
Note over Frontend,Controller: Model discovery
Frontend->>Controller: POST /freescoutgpt/litellm-models (baseUrl, apiKey)
Controller->>Cache: lookup(baseUrl+apiKey)
alt cache hit
Cache-->>Controller: models
else cache miss
Controller->>LiteLLM: GET {baseUrl}/v1/models (Bearer apiKey?)
LiteLLM-->>Controller: { data: [...] }
Controller->>Cache: store(models, 10m)
end
Controller-->>Frontend: { data: [...] }
Frontend->>Frontend: populate model dropdown
end
sequenceDiagram
participant Frontend
participant Controller
participant LiteLLM
participant DB
rect rgba(100, 150, 200, 0.5)
Note over Frontend,Controller: Content generation (LiteLLM branch)
Frontend->>Controller: POST /generate (query, litellm_enabled=true, settings)
alt use_responses_api && litellm_enabled
Controller->>LiteLLM: POST {baseUrl}/v1/responses (prompt + context)
LiteLLM-->>Controller: { output: [...] } / error
Controller->>Controller: extract first text output
else litellm_enabled
Controller->>LiteLLM: POST {baseUrl}/v1/chat/completions (messages)
LiteLLM-->>Controller: { choices: [{ message: { content } }] } / error
Controller->>Controller: extract content
end
Controller->>DB: persist answer to thread chatgpt history
Controller-->>Frontend: { query, answer }
end
Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly Related PRs
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
Http/Controllers/FreeScoutGPTController.php (1)
462-469: Consider validatinglitellm_base_urlwhen LiteLLM is enabled.If
litellm_enabledis true butlitellm_base_urlis empty or null, the Guzzle request will fail with an unclear error. Consider adding a guard clause with a user-friendly error message.💡 Proposed validation
// LiteLLM Proxy API: use if enabled, before direct OpenAI if (!empty($settings->litellm_enabled)) { \Log::info('Using LiteLLM Proxy for answers'); $litellmBaseUrl = rtrim($settings->litellm_base_url ?? '', '/'); + if (empty($litellmBaseUrl)) { + return Response::json([ + 'query' => $request->get('query') ?? '', + 'answer' => 'LiteLLM is enabled but no base URL is configured. Please configure the LiteLLM proxy URL in settings.' + ], 200); + } $litellmApiKey = $settings->litellm_api_key; $litellmModel = $settings->litellm_model;🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@Http/Controllers/FreeScoutGPTController.php` around lines 462 - 469, When LiteLLM is enabled (check $settings->litellm_enabled) validate that $settings->litellm_base_url is present and a non-empty valid URL before using rtrim() (litellmBaseUrl); in FreeScoutGPTController add a guard clause that returns a clear error response (or throws a BadRequest/ValidationException) if litellm_base_url is empty or malformed, and log the validation failure so the subsequent Guzzle request is never made with an empty base URL.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@Http/Controllers/FreeScoutGPTController.php`:
- Around line 550-561: The code assigns $customerEmail inside the
FreeScoutGPTController when $settings->client_data_enabled is true but never
uses it; remove the unused assignment or incorporate it into the system message.
Update the block inside the conditional (references:
$settings->client_data_enabled, $customerEmail, $customerName,
$conversationSubject) so either (a) delete the line that sets $customerEmail =
$request->get("customer_email"); or (b) include the email in the messages[]
system content (e.g., add a "Customer email is: :email" placeholder) and pass
$customerEmail into the translation array, ensuring no unused variables remain.
- Around line 250-290: The getAvailableLitellmModels method currently trusts the
user-supplied $baseUrl and performs server-side requests; validate and sanitize
$baseUrl before using it: first ensure filter_var($baseUrl, FILTER_VALIDATE_URL)
passes and enforce scheme === 'https' (or restrict to an allowlist of domains if
required) by parsing with parse_url; then resolve the host (e.g., dns_get_record
or gethostbyname) and reject addresses in private/internal ranges (check with
ip2long and compare against RFC1918 ranges and link-local/metadata IPs like
169.254.0.0/16); if validation fails, return a 400 error and do not perform the
Guzzle request or cache the result (references: getAvailableLitellmModels,
$baseUrl, $apiKey, $cacheKey).
In `@Public/js/settings.js`:
- Around line 283-284: The code sets litellmModelSelect.innerHTML using
untrusted data.error which allows XSS; change this to create an Option element
(or set option via textContent) and assign its text/value safely instead of
using innerHTML — locate the branch that uses litellmModelSelect.innerHTML with
data.error and replace it with creating an option node (e.g., new Option or
document.createElement('option')), set its textContent (or option.text) to a
sanitized/error string and its value to an empty or escaped value, then append
that option to litellmModelSelect so the error message is rendered as plain text
not HTML.
---
Nitpick comments:
In `@Http/Controllers/FreeScoutGPTController.php`:
- Around line 462-469: When LiteLLM is enabled (check
$settings->litellm_enabled) validate that $settings->litellm_base_url is present
and a non-empty valid URL before using rtrim() (litellmBaseUrl); in
FreeScoutGPTController add a guard clause that returns a clear error response
(or throws a BadRequest/ValidationException) if litellm_base_url is empty or
malformed, and log the validation failure so the subsequent Guzzle request is
never made with an empty base URL.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c8400f9d-fbf6-4f67-90f1-e3dc775b9a70
📒 Files selected for processing (6)
Database/Migrations/2025_06_15_000001_add_litellm_fields_to_freescoutgpt_table.phpEntities/GPTSettings.phpHttp/Controllers/FreeScoutGPTController.phpHttp/routes.phpPublic/js/settings.jsResources/views/settings.blade.php
… var - Validate litellm_base_url with filter_var and scheme check to prevent SSRF - Use textContent instead of innerHTML for error messages to prevent XSS - Remove unused $customerEmail variable in LiteLLM chat completions path
There was a problem hiding this comment.
🧹 Nitpick comments (2)
Http/Controllers/FreeScoutGPTController.php (2)
590-593: Same error sanitization concern applies here.Similar to the Responses API error handling, the Chat Completions path also exposes raw exception messages to users.
🛡️ Proposed fix
} catch (\Exception $e) { - \Log::error('LiteLLM Chat Completions Error: ' . $e->getMessage()); - $answerText = $e->getMessage(); + \Log::error('LiteLLM Chat Completions Error: ' . $e->getMessage()); + $answerText = __('An error occurred while processing your request. Please try again later.'); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@Http/Controllers/FreeScoutGPTController.php` around lines 590 - 593, The catch block in FreeScoutGPTController that currently logs and returns raw exception text (the catch(\Exception $e) handling that sets $answerText = $e->getMessage() and calls \Log::error(...)) must avoid exposing internal error details to users: change it to log the full exception (including stack/trace) to the server logs using the exception object, and set $answerText to a generic sanitized message (e.g., "An internal error occurred while generating the response") before returning; ensure the \Log::error call references the exception (or its trace) for debugging while the user-facing $answerText contains no raw $e->getMessage() content.
529-546: Consider sanitizing error messages before returning to users.The exception handling exposes raw error messages (including
$proxyErrorfrom the proxy response body) directly to users. This could leak internal infrastructure details such as internal hostnames, IP addresses, or configuration information from the LiteLLM proxy.🛡️ Proposed fix to sanitize error output
} - $answerText = $proxyError ?: $errorMsg; + $answerText = __('An error occurred while processing your request. Please try again later.'); \Log::error('Error on LiteLLM Responses API Request: ' . $answerText); + \Log::error('LiteLLM Responses API raw error: ' . ($proxyError ?: $errorMsg)); return Response::json([ 'query' => $userQuery ?? '', 'answer' => $answerText🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@Http/Controllers/FreeScoutGPTController.php` around lines 529 - 546, The catch block in FreeScoutGPTController currently returns raw exception details ($proxyError / $errorMsg) to users; change it to sanitize and redact sensitive data before returning while logging the full error internally. In the catch handler for the method that contains the current try/catch, capture and log the full exception (\Log::error($e) or similar), then create a sanitized message by stripping/obfuscating hostnames, IP addresses and URLs from $proxyError/$errorMsg (use a regex to replace IPs, hostnames and http(s) URLs with “[REDACTED]”), truncate to a safe max length, and if the result is empty return a generic user-facing message like “An internal error occurred; please try again.” Finally, set $answerText to this sanitized/generic value for the JSON response while ensuring full details remain in the logs.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@Http/Controllers/FreeScoutGPTController.php`:
- Around line 590-593: The catch block in FreeScoutGPTController that currently
logs and returns raw exception text (the catch(\Exception $e) handling that sets
$answerText = $e->getMessage() and calls \Log::error(...)) must avoid exposing
internal error details to users: change it to log the full exception (including
stack/trace) to the server logs using the exception object, and set $answerText
to a generic sanitized message (e.g., "An internal error occurred while
generating the response") before returning; ensure the \Log::error call
references the exception (or its trace) for debugging while the user-facing
$answerText contains no raw $e->getMessage() content.
- Around line 529-546: The catch block in FreeScoutGPTController currently
returns raw exception details ($proxyError / $errorMsg) to users; change it to
sanitize and redact sensitive data before returning while logging the full error
internally. In the catch handler for the method that contains the current
try/catch, capture and log the full exception (\Log::error($e) or similar), then
create a sanitized message by stripping/obfuscating hostnames, IP addresses and
URLs from $proxyError/$errorMsg (use a regex to replace IPs, hostnames and
http(s) URLs with “[REDACTED]”), truncate to a safe max length, and if the
result is empty return a generic user-facing message like “An internal error
occurred; please try again.” Finally, set $answerText to this sanitized/generic
value for the JSON response while ensuring full details remain in the logs.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 0e14ba61-96ca-4d77-8d91-1e6cd7364cc9
📒 Files selected for processing (2)
Http/Controllers/FreeScoutGPTController.phpPublic/js/settings.js
🚧 Files skipped from review as they are similar to previous changes (1)
- Public/js/settings.js
Author
|
Any news when if this will be merged? |
Owner
|
@bigtcze A quick review has me asking, could this be used for almost any custom LLM URL? Not just litellm? Let me review and test this more... and I can look at merging after that. |
Author
|
Thank you @presswizards I designed it specificaly for litellm, which is my usecase I'd like to have. I havent consirened any other custom LLM URL. I know, that litellm supports high number of providers. I currently use modern models from Google, OpenAI and Antropic, which is fine with litellm. This change basicaly delegates LLM calls from your project to deditated project, which should help you be more up-to-date with very fast changing world of AI and new LLMs appearing nearly every month. |
Owner
|
@bigtcze I am just wondering, if the options: litellm_base_url could be named more generic since they are OpenAI-compatible I believe: llmproxy_base_url I think it would work for others besides LiteLLM, like for OpenRouter, PortKey, BitFrost, and other OpenAI compatible AI gateways or proxies? Because I'd rather have it more open instead of trying to support each gateway or proxy as named options separately... perhaps even rework the existing OpenAI Chat Completions option to be "OpenAI-Compatible Gateway" instead, with an added URL field? It could be as simple as using: llm_base_url or llm_gateway_base_url and pull in the models based on what's returned from the API Key. |
Author
|
Ok, give me few days, I'll update it |
Replace the separate LiteLLM proxy option with a single API Base URL field that works with any OpenAI-compatible gateway or proxy (OpenRouter, PortKey, LiteLLM, BitFrost, etc.). When the URL is left empty, requests go directly to OpenAI as before. - Merge OpenAI + LiteLLM code paths into one unified Guzzle-based flow - Add api_base_url field with DB migration - Use 'system' role for custom gateways, 'developer'/'user' for OpenAI - Use max_tokens (universal) for chat completions compatibility - Make API key and Authorization header optional for local proxies - Strip trailing /v1 from base URL to prevent path doubling - Remove LiteLLM UI section, route, controller method, and migration - Remove unused Tectalic OpenAI dependency - Model dropdown now fetches from whatever endpoint is configured
bigtcze
changed the title
Author
|
Ok there it is :) |
Author
|
Hello @presswizards did you have a time to check it? |
Owner
|
@bigtcze Oh very nice, sorry I missed the updated commits... let me test this out and ensure OpenAI, Infomaniak, and now the new Base URL options all work as intended. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworked the original LiteLLM proxy PR into a universal OpenAI-Compatible Gateway approach, as suggested by @presswizards. Instead of adding LiteLLM as a named option, this adds a single API Base URL field that works with any OpenAI-compatible gateway or proxy — OpenRouter, PortKey, LiteLLM, BitFrost, or any other.
What changed
API Base URLfield — leave empty for direct OpenAI, set a URL to route through any gatewaydeveloper/userroles for direct OpenAI,systemrole for gateways (universal compatibility)max_tokensinstead ofmax_output_tokensin Chat Completions for broad gateway support/v1path stripping — prevents URL doubling when users enterhttps://openrouter.ai/api/v1(code appends/v1/...itself)/v1/modelsHow it works
https://openrouter.ai/apihttp://localhost:4000Migration
New DB migration adds
api_base_urlcolumn. No breaking changes for existing OpenAI-direct users.