feat(llm): support authenticated OpenAI-compatible endpoints (openai-compatible) - #276
feat(llm): support authenticated OpenAI-compatible endpoints (openai-compatible)#276JunYanForFunny wants to merge 2 commits into
openai-compatible)#276Conversation
📝 WalkthroughWalkthroughThe LLM integration now supports authenticated OpenAI-compatible endpoints with configurable base URLs and models. CLI validation and key storage accept the provider. Local synthesis uses an 8,000-token default. ChangesOpenAI-compatible provider
Local synthesis budget
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant LLMSelector
participant OpenAICompatibleAdapter
participant OpenAIClient
participant OpenAICompatibleEndpoint
LLMSelector->>OpenAICompatibleAdapter: select openai-compatible and model
OpenAICompatibleAdapter->>OpenAIClient: configure API key and base URL
OpenAICompatibleAdapter->>OpenAIClient: send completion request
OpenAIClient->>OpenAICompatibleEndpoint: authenticated request
OpenAICompatibleEndpoint-->>OpenAIClient: completion response
OpenAIClient-->>OpenAICompatibleAdapter: text or extracted JSON
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/configuration.md`:
- Line 107: Correct the WIGOLO_LLM_BASE_URL documentation so openai-compatible
is not shown with the Ollama default; state that it requires an explicitly
configured OpenAI-compatible /v1 endpoint, or separate the Ollama and
openai-compatible provider entries with their respective defaults.
In `@src/cli/tui/flags.ts`:
- Line 143: Update the VALID_PROVIDERS allowlist to include 'groq', preserving
the existing provider entries and validation behavior so --provider=groq is
accepted by init.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 91e0c1ce-026b-4d59-aa3c-d5e9fedc5682
📒 Files selected for processing (10)
docs/configuration.mdsrc/cli/init.tssrc/cli/tui/flags.tssrc/integrations/cloud/llm/model-select.tssrc/integrations/cloud/llm/select.tssrc/integrations/cloud/llm/text-adapters.tssrc/integrations/cloud/llm/types.tssrc/research/synthesis-local.tssrc/security/key-store.tstests/unit/extraction/llm/types.test.ts
| } | ||
|
|
||
| const VALID_PROVIDERS = ['anthropic', 'openai', 'gemini', 'ollama'] as const; | ||
| const VALID_PROVIDERS = ['anthropic', 'openai', 'gemini', 'ollama', 'openai-compatible'] as const; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Restore groq in VALID_PROVIDERS.
Line 143 rejects wigolo init --provider=groq. The init usage text and provider contract support groq. Add it to this allowlist.
Proposed fix
-const VALID_PROVIDERS = ['anthropic', 'openai', 'gemini', 'ollama', 'openai-compatible'] as const;
+const VALID_PROVIDERS = ['anthropic', 'openai', 'gemini', 'groq', 'ollama', 'openai-compatible'] as const;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const VALID_PROVIDERS = ['anthropic', 'openai', 'gemini', 'ollama', 'openai-compatible'] as const; | |
| const VALID_PROVIDERS = ['anthropic', 'openai', 'gemini', 'groq', 'ollama', 'openai-compatible'] as const; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/cli/tui/flags.ts` at line 143, Update the VALID_PROVIDERS allowlist to
include 'groq', preserving the existing provider entries and validation behavior
so --provider=groq is accepted by init.
bab42f5 to
68118de
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/integrations/cloud/llm/openai.ts`:
- Around line 64-65: Update extractWithLLM and the OpenAI-compatible adapter
flow to import and use resolveModel, passing the resolved
WIGOLO_LLM_MODEL_OPENAI_COMPATIBLE value or its existing fallback as
modelOverride. Ensure the selected model is used for the OpenAI request and
reflected in the cache identity instead of provider:default.
- Around line 72-79: Update callOpenAICompatible to avoid unconditionally
requiring Structured Outputs: add endpoint capability handling and use the
provider-supported response_format, including json_object for endpoints such as
DeepSeek, with local opts.jsonSchema validation when needed; otherwise reject
endpoints lacking Structured Outputs with a clear error.
In `@src/integrations/cloud/llm/text-adapters.ts`:
- Around line 183-189: Update callOpenAICompatibleText’s chat.completions.create
request to use max_tokens for the generic compatible token limit, while
retaining max_completion_tokens in provider-specific adapters that support it.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 67a1eade-c386-42c4-8e53-bbafede59039
📒 Files selected for processing (12)
docs/configuration.mdsrc/cli/init.tssrc/cli/tui/flags.tssrc/extraction/llm-fallback.tssrc/integrations/cloud/llm/model-select.tssrc/integrations/cloud/llm/openai.tssrc/integrations/cloud/llm/select.tssrc/integrations/cloud/llm/text-adapters.tssrc/integrations/cloud/llm/types.tssrc/research/synthesis-local.tssrc/security/key-store.tstests/unit/extraction/llm/types.test.ts
🚧 Files skipped from review as they are similar to previous changes (9)
- src/security/key-store.ts
- src/cli/init.ts
- src/integrations/cloud/llm/types.ts
- src/integrations/cloud/llm/select.ts
- docs/configuration.md
- tests/unit/extraction/llm/types.test.ts
- src/research/synthesis-local.ts
- src/integrations/cloud/llm/model-select.ts
- src/cli/tui/flags.ts
| const client = new OpenAI({ apiKey, baseURL }); | ||
| const model = opts.modelOverride ?? DEFAULT_MODEL; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
ast-grep outline src/integrations/cloud/llm --items all --type function
rg -n -C 4 'WIGOLO_LLM_MODEL_OPENAI_COMPATIBLE|modelOverride|callOpenAICompatible|modelId' \
src/integrations/cloud/llm src/extraction src/config.tsRepository: KnockOutEZ/wigolo
Length of output: 16733
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n=== openai.ts relevant implementation ===\n'
cat -n src/integrations/cloud/llm/openai.ts | sed -n '1,120p'
printf '\n=== llm-fallback.ts relevant implementation ===\n'
cat -n src/extraction/llm-fallback.ts | sed -n '1,150p'
printf '\n=== model-select.ts relevant implementation ===\n'
cat -n src/integrations/cloud/llm/model-select.ts | sed -n '1,80p'
printf '\n=== all callOpenAICompatible call sites ===\n'
rg -n -C 3 'callOpenAICompatible\(' src
printf '\n=== ExtractWithLLM resolved provider/model call sites ===\n'
rg -n -C 4 'extractWithLLM\(' srcRepository: KnockOutEZ/wigolo
Length of output: 12696
Use the selected extraction model for OpenAI-compatible fallback requests.
extractWithLLM resolves the provider and uses provider:default for cache identity, but the adapter receives no modelOverride. For OpenAI-compatible extraction, import resolveModel and pass WIGOLO_LLM_MODEL_OPENAI_COMPATIBLE/fallback values so the OpenAI-compatible model is requested and cached under a model-stable ID.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/integrations/cloud/llm/openai.ts` around lines 64 - 65, Update
extractWithLLM and the OpenAI-compatible adapter flow to import and use
resolveModel, passing the resolved WIGOLO_LLM_MODEL_OPENAI_COMPATIBLE value or
its existing fallback as modelOverride. Ensure the selected model is used for
the OpenAI request and reflected in the cache identity instead of
provider:default.
| response_format: { | ||
| type: 'json_schema', | ||
| json_schema: { | ||
| name: 'extract', | ||
| schema: opts.jsonSchema, | ||
| strict: true, | ||
| }, | ||
| }, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
ast-grep outline src/integrations/cloud/llm/openai.ts --match callOpenAICompatible --view expanded
rg -n -C 4 --glob '*.ts' \
'callOpenAICompatible|response_format|json_schema|json_object' src testsRepository: KnockOutEZ/wigolo
Length of output: 9584
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== openai.ts relevant implementation =="
cat -n src/integrations/cloud/llm/openai.ts | sed -n '1,130p'
echo
echo "== text adapters relevant implementation =="
cat -n src/integrations/cloud/llm/text-adapters.ts | sed -n '150,220p'
echo
echo "== all WIGOLO_LLM_BASE_URL / openai-compatible references =="
rg -n -C 3 'WIGOLO_LLM_BASE_URL|openai-compatible|OpenAI-compatible|DeepSeek|response_format|json_schema|json_object' src tests README.md docs 2>/dev/null || true
echo
echo "== validate.ts =="
cat -n src/integrations/cloud/llm/validate.ts | sed -n '1,160p'Repository: KnockOutEZ/wigolo
Length of output: 42977
🌐 Web query:
DeepSeek API create chat completion response_format json_object response_format documentation
💡 Result:
The DeepSeek API supports a JSON output mode that ensures the model generates valid JSON strings [1][2]. To use this feature, you must configure the response_format parameter in your Chat Completions API request [1][3]. Key requirements and implementation details: Parameter Configuration: Set the response_format parameter to {"type": "json_object"} [1][2]. The default value for this parameter is "text" [1]. Prompt Guidance: It is mandatory to explicitly instruct the model to produce JSON in your system or user message [1][2]. You should include the word "json" in your prompt and provide an example of the desired JSON structure [2]. Failing to provide this instruction may cause the model to generate unintended whitespace or experience processing delays, as it will attempt to satisfy the JSON requirement without clear formatting guidance [1][4]. Token Management: Ensure the max_tokens parameter is set to an appropriate limit to prevent the generated JSON string from being truncated prematurely [2][3]. The JSON Output feature is designed to be compatible with the OpenAI API format, facilitating integration into existing workflows [3][5]. When enabled, the model output can be parsed directly as a JSON object [1][2]. Additionally, be aware that the API may occasionally return empty content when using this mode, in which case modifying your prompt may help [2].
Citations:
- 1: https://api-docs.deepseek.com/api/create-chat-completion/
- 2: https://api-docs.deepseek.com/guides/json_mode/
- 3: https://api-docs.deepseek.com/news/news0725/
- 4: https://docs.spring.io/spring-ai/docs/current/api/org/springframework/ai/deepseek/api/ResponseFormat.html
- 5: https://api-docs.deepseek.com/news/news0725
Avoid requiring Structured Outputs from every OpenAI-compatible endpoint.
callOpenAICompatible always sends response_format.type = "json_schema" with strict: true, but documented OpenAI-compatible providers such as DeepSeek expose JSON mode as response_format.type = "json_object". This makes the advertised provider incompatible for extraction. Add endpoint capabilities and either use json_object with local schema validation or reject endpoints that do not support Structured Outputs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/integrations/cloud/llm/openai.ts` around lines 72 - 79, Update
callOpenAICompatible to avoid unconditionally requiring Structured Outputs: add
endpoint capability handling and use the provider-supported response_format,
including json_object for endpoints such as DeepSeek, with local opts.jsonSchema
validation when needed; otherwise reject endpoints lacking Structured Outputs
with a clear error.
| const response = await client.chat.completions.create( | ||
| { | ||
| model: opts.model, | ||
| max_completion_tokens: opts.maxTokens ?? DEFAULT_MAX_TOKENS, | ||
| messages: [{ role: 'user', content: opts.prompt }], | ||
| }, | ||
| { signal: opts.signal }, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
ast-grep outline src/integrations/cloud/llm/text-adapters.ts \
--match callOpenAICompatibleText --view expanded
rg -n -C 4 --glob '*.ts' \
'callOpenAICompatibleText|max_completion_tokens|max_tokens|openai-compatible' src testsRepository: KnockOutEZ/wigolo
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '80,105p' src/integrations/cloud/llm/text-adapters.ts
sed -n '140,170p' src/integrations/cloud/llm/text-adapters.ts
sed -n '165,192p' src/integrations/cloud/llm/text-adapters.ts
echo "Package openai version:"
node - <<'JS'
try {
const fs = require('fs');
const p = JSON.parse(fs.readFileSync('package.json', 'utf8'));
for (const k of ['dependencies','devDependencies','packageManager']) {
console.log(k, p[k] || '<not set>');
}
} catch (e) { console.log('package.json unavailable:', e.message); }
JS
echo "Lockfile references:"
(grep -n '"openai"' package-lock.json 2>/dev/null || true)
(grep -n '"openai"' pnpm-lock.yaml 2>/dev/null || true)
(grep -n 'openai@' yarn.lock 2>/dev/null || true) || trueRepository: KnockOutEZ/wigolo
Length of output: 4432
🌐 Web query:
DeepSeek API chat completions max_tokens max_completion_tokens documentation
💡 Result:
The DeepSeek API uses max_tokens as the parameter to limit the maximum number of tokens generated in a chat completion [1][2][3]. It does not support the max_completion_tokens parameter used by some other providers (such as OpenAI's o-series models) [4][5]. If an application sends max_completion_tokens to the DeepSeek API, it will typically be ignored or rejected, which can lead to issues where token limits are not respected [5]. To ensure compatibility with the DeepSeek API, applications should remap any max_completion_tokens inputs to max_tokens [4][5]. Key details regarding max_tokens: - Function: Caps the number of tokens generated in the completion [1][3]. - Context Limit: The total number of input tokens plus generated tokens remains subject to the model's overall context window limit [1][3]. - Limits: While the default is often 4096, some configurations or beta endpoints allow for higher limits (e.g., up to 8192) [2][6]. - Compatibility: When using tools or SDKs designed for OpenAI/Anthropic, developers often need to explicitly handle this field mapping to maintain compatibility [7][4][5].
Citations:
- 1: https://api-docs.deepseek.com/api/create-chat-completion
- 2: https://github.com/api-evangelist/deepseek/blob/main/openapi/deepseek-chat-completion-api-openapi.yml
- 3: https://chat-deep.ai/guide/create-chat-completion/
- 4: mozilla-ai/any-llm@34adffe
- 5: fix: use max_tokens instead of max_completion_tokens for DeepSeek com… ZhiYi-R/moon-bridge#88
- 6: https://api-docs.deepseek.com/news/news0725/
- 7: https://api-docs.deepseek.com/quick_start/agent_integrations/oh_my_pi/
Use max_tokens for the generic compatible limit.
callOpenAICompatibleText targets arbitrary WIGOLO_LLM_BASE_URL backends, including DeepSeek-compatible endpoints. These backends may reject or ignore max_completion_tokens, so set the OpenAI-compatible token cap with max_tokens; keep max_completion_tokens for adapters that target providers that support it.
Proposed fix
- max_completion_tokens: opts.maxTokens ?? DEFAULT_MAX_TOKENS,
+ max_tokens: opts.maxTokens ?? DEFAULT_MAX_TOKENS,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const response = await client.chat.completions.create( | |
| { | |
| model: opts.model, | |
| max_completion_tokens: opts.maxTokens ?? DEFAULT_MAX_TOKENS, | |
| messages: [{ role: 'user', content: opts.prompt }], | |
| }, | |
| { signal: opts.signal }, | |
| const response = await client.chat.completions.create( | |
| { | |
| model: opts.model, | |
| max_tokens: opts.maxTokens ?? DEFAULT_MAX_TOKENS, | |
| messages: [{ role: 'user', content: opts.prompt }], | |
| }, | |
| { signal: opts.signal }, |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/integrations/cloud/llm/text-adapters.ts` around lines 183 - 189, Update
callOpenAICompatibleText’s chat.completions.create request to use max_tokens for
the generic compatible token limit, while retaining max_completion_tokens in
provider-specific adapters that support it.
Add a generic OpenAI-compatible LLM provider so wigolo can synthesize against authenticated remote OpenAI-compatible endpoints (OpenRouter, DeepSeek, SensNova, self-hosted vLLM, etc.) using an API key. - types: extend LLMProvider union with 'openai-compatible' - select: register provider order + WIGOLO_LLM_API_KEY env; make it explicit-only so a bare WIGOLO_LLM_API_KEY is never auto-detected (preserves the KnockOutEZ#102 contract: the generic key is ambiguous without an explicit provider) - model-select: default model + WIGOLO_LLM_MODEL_OPENAI_COMPATIBLE env - text-adapters: callOpenAICompatibleText routes to WIGOLO_LLM_BASE_URL with the key as a Bearer token (unlike the keyless ollama/custom path) - openai.ts: callOpenAICompatible JSON-extraction adapter (same base_url) - llm-fallback/key-store/select/init/flags: recognize the provider end-to-end - docs/configuration.md: document the new provider + env vars - tests: update LLMProvider union assertion, add explicit-selection test, export callOpenAICompatible in the openai.js mock
synthesis-local defaulted to 3000 completion tokens. Reasoning-capable models spend part of that budget on a hidden 'reasoning' field and can return empty 'content', which the adapter rejects and wigolo downgrades to the heuristic fallback. Bump the budget to 8000 so reasoning models have headroom to emit content.
68118de to
be60305
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
1 participant
Summary
Add a generic
openai-compatibleLLM provider so wigolo can synthesize against authenticated remote OpenAI-compatible endpoints (OpenRouter, DeepSeek, SensNova, self-hosted vLLM, etc.) using an API key.Today the only way to point wigolo at a custom OpenAI-compatible URL is the
ollama/ custom-backend path, which is keyless (run.jssends noAuthorizationheader — confirmed in source). So any authenticated endpoint 401s. This PR adds a first-class provider that sends the key as a Bearer token toWIGOLO_LLM_BASE_URL.This is the generic capability that PR #254 (MiniMax) hard-codes per-vendor — one type replaces "one PR per provider".
Changes
types.ts: extendLLMProviderunion with'openai-compatible'.select.ts: register provider order +WIGOLO_LLM_API_KEYenv. Explicit-only: a bareWIGOLO_LLM_API_KEYis never auto-detected (preserves the WIGOLO_LLM_API_KEY env var is not recognized — should accept GOOGLE_API_KEY for Gemini provider #102 contract — the generic key is ambiguous without an explicit provider).WIGOLO_LLM_PROVIDER=openai-compatibleis required.model-select.ts: default model +WIGOLO_LLM_MODEL_OPENAI_COMPATIBLEenv.text-adapters.ts:callOpenAICompatibleTextroutes togetConfig().llmBaseUrlwith the key as a Bearer token.openai.ts:callOpenAICompatibleJSON-extraction adapter (same base_url) for thellm-fallbackpath.llm-fallback.ts/key-store.ts/init.ts/flags.ts: recognize the provider end-to-end.docs/configuration.md: document the new provider + env vars.synthesis-local.ts: raise the local-synthesis completion-token budget 3000 → 8000, so reasoning-capable models (which spend part of the budget on a hiddenreasoningfield) have headroom to emitcontentinstead of being downgraded to the heuristic fallback.LLMProviderunion assertion, add an explicit-selection test, exportcallOpenAICompatiblein the openai.js mock.Config
Testing
npm run lint(tsc --noEmit): passes.GOOGLE_API_KEY/WIGOLO_LLM_API_KEYin the process):select.test.ts,llm-select-seam.test.ts,llm-fallback.test.ts(38 tests),synthesis-local.test.ts(17 tests).tests/unit/repl/shell*.test.ts(NDJSON/history tests require a TTY) andsynthesis-local.test.ts's "throws when local LLM not configured" when a realGEMINI_API_KEYis present in the env (the test's setup only clearsWIGOLO_LLM_PROVIDER/WIGOLO_LLM_MODEL, so the gate sees Gemini configured and attempts a real call). Both pass once the ambient key is unset.Risks / notes
api.openai.comfor the fixedopenaiprovider and ignoresWIGOLO_LLM_BASE_URL; this provider is the supported way to override the base URL with a key.dist/is gitignored and rebuilt at publish (tsup && tsc), so this PR only touchessrc/.Summary by CodeRabbit
openai-compatibleas a selectable provider during setup and configuration.gpt-4o-mini.