Fix/llxprt tool calls and sse streaming#66
Open
marmarjohnson wants to merge 4 commits intoAmm1rr:masterfrom
Open
Fix/llxprt tool calls and sse streaming#66marmarjohnson wants to merge 4 commits intoAmm1rr:masterfrom
marmarjohnson wants to merge 4 commits intoAmm1rr:masterfrom
Conversation
Fixes two issues preventing llxprt from working correctly with
WebAI-to-API via its gemini and openai provider profiles.
## Fix 1: Gemini provider — "Incomplete JSON segment" error (test.json)
The @google/genai SDK always calls streamGenerateContent and expects
SSE format (text/event-stream with `data: {...}` lines). The endpoint
was returning plain JSON, causing the SDK to fail parsing.
Changes in google_generative.py:
- Route changed from `/v1beta/models/{model}` to
`/v1beta/models/{model_path:path}` to correctly capture the
`:streamGenerateContent` action suffix in the URL
- Detects the action (generateContent vs streamGenerateContent) and
returns a StreamingResponse with text/event-stream content type
- Accepts tools (functionDeclarations), systemInstruction, and
generationConfig fields that were previously causing 422 errors
- Injects tool definitions as a system prompt and parses functionCall
JSON from the response, returning it in Gemini native part format
## Fix 2: OpenAI provider — tool calls not recognized (webai.json)
The tools field in OpenAI chat completion requests was silently
ignored and responses never contained tool_calls.
Changes in chat.py:
- OpenAIChatRequest now accepts tools and tool_choice fields
- Tool definitions are injected as a system prompt instructing Gemini
to respond as {"tool_call": {"name": ..., "arguments": {...}}}
- Handles role:tool messages (tool results) and assistant messages
with tool_calls in conversation history
- Parses Gemini response text for tool call JSON and returns proper
OpenAI tool_calls format with finish_reason: "tool_calls"
## Fix 3: Schema — Part model made flexible (request.py)
Part.text was required (str), causing 422 errors for Gemini SDK
requests that include functionCall or functionResponse parts.
Made text optional and added functionCall and functionResponse fields.
Also added Tool and FunctionDeclaration models to support the
tools field in GoogleGenerativeRequest.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Regex patterns with [^{}]* failed when tool arguments contained nested
objects (e.g. {"dir_path": "..."}). Replace with json.JSONDecoder.raw_decode
which scans for { and handles arbitrary nesting depth. Also relax model
fields from GeminiModels enum to str to allow any model name.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tool calls weren't working for me for either endpoint. Note, I also have a commit to the script update.sh that updates and starts the job in the background if its not already running (running on Ubuntu 24.04)