feat(backend): surface self-hosted reasoning via @ai-sdk/openai-compatible#262
Open
MesoX wants to merge 2 commits into
Open
feat(backend): surface self-hosted reasoning via @ai-sdk/openai-compatible#262MesoX wants to merge 2 commits into
MesoX wants to merge 2 commits into
Conversation
Chat-completions mode (apiMode=chat) is used in practice only against self-hosted/company OpenAI-compatible servers (vLLM, SGLang, llama.cpp, TGI). Those expose model thinking in a `reasoning_content` field that `@ai-sdk/openai`'s chat model silently drops, so the existing reasoning UI never received it. Route chat-mode language models through `@ai-sdk/openai-compatible`, which reads `reasoning_content` natively and emits reasoning stream parts. Embeddings and native search tools stay on the OpenAI SDK. Responses mode is unchanged. Real OpenAI in chat mode is unaffected (it returns no reasoning_content). Leaves supportsStructuredOutputs at default false: a requested JSON schema downgrades to json_object mode (AI SDK still validates client-side) to stay safe across servers lacking json_schema support. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`@ai-sdk/openai-compatible` only sends `stream_options.include_usage` when `includeUsage` is set, so self-hosted servers (vLLM, SGLang, …) returned no token usage on the streaming path — surfacing as In:0/Out:0 in the UI. Non-streaming (e.g. compaction summarize) was unaffected. Set `includeUsage: true` so streamed turns carry token counts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Chat-completions mode (
apiMode=chat) on anOpenAIprovider is, in practice, only used against self-hosted / company OpenAI-compatible servers (vLLM, SGLang, llama.cpp, TGI, LM Studio) — real OpenAI is driven via the Responses API. Those servers expose the model's thinking in areasoning_contentfield, but@ai-sdk/openai's chat model only parsesrole/content/tool_callsand silently drops it. So the existing reasoning UI never received anything from them.This routes chat-mode language models through
@ai-sdk/openai-compatible, which readsreasoning_content(andreasoning) natively and emits reasoning stream parts — rendered by the existing collapsible "Thinking…" block. No frontend changes.Changes
@ai-sdk/openai-compatibledependency (depends on the same@ai-sdk/provider@3as the other adapters).provider.ts: forOpenAIproviders withapiMode==="chat", build the language model viacreateOpenAICompatible(...). Embeddings and native search tools stay on the OpenAI SDK. Responses mode is unchanged.includeUsage: trueso streamed turns carrystream_options.include_usage— without it self-hosted servers return no token usage on the streaming path (shows as In:0 / Out:0).Behavior notes
reasoning_content, so no reasoning parts are emitted.supportsStructuredOutputsis left at its default (false): a requested JSON schema downgrades tojson_objectmode (the AI SDK still validates client-side), which is safe across servers that don't supportjson_schemaresponse_format. This only affects the title/tag generation call.Testing
typecheck,lint, and theproviderunit tests pass.🤖 Generated with Claude Code