Skip to content

fix(llm): add conditional max_tokens parameter naming per provider#340

Merged
ajianaz merged 1 commit into
developfrom
fix/max-tokens-param-conditional
Jul 7, 2026
Merged

fix(llm): add conditional max_tokens parameter naming per provider#340
ajianaz merged 1 commit into
developfrom
fix/max-tokens-param-conditional

Conversation

@ajianaz

@ajianaz ajianaz commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

LLM API providers use different parameter names for max output tokens:

Provider Parameter
OpenAI (legacy) max_tokens
OpenAI (new) max_completion_tokens
Anthropic max_tokens
Google Gemini / Vertex max_output_tokens
Some proxies map max_tokensmax_output_tokens and then reject

cora-cli previously hardcoded max_tokens everywhere. When a provider doesn't support it, we get HTTP 400: Unsupported parameter: max_output_tokens.

Solution

Add max_tokens_param to the llm section in .cora.yaml. Supports:

  • auto (default): detect from provider name.
    • gemini / google / vertexmax_output_tokens
    • Everything else → max_tokens
  • max_tokens, max_output_tokens, max_completion_tokens: explicit override

Config example

llm:
  max_tokens_param: auto       # default — detect from provider
  # max_tokens_param: max_output_tokens  # explicit override

Changes

  • src/config/schema.rs: Added max_tokens_param: Option<String> to LlmSection and max_tokens_param: String to Config (default: "auto")
  • src/engine/types.rs: Added max_tokens_param: String to LLMConfig (default: "max_tokens")
  • src/config/loader.rs: Added resolve_max_tokens_param() for auto-detection; resolved value passed to LLMConfig
  • src/engine/llm.rs: Replaced hardcoded "max_tokens" JSON key with dynamic config.max_tokens_param in both streaming and non-streaming request paths
  • src/commands/config_cmd.rs: Display max_tokens_param in cora config output

Tests added

  • test_resolve_max_tokens_param_auto_* — auto-detection for known providers (gemini, google, vertex, openai, anthropic)
  • test_resolve_max_tokens_param_explicit_override — explicit value wins over auto
  • test_chat_request_uses_max_output_tokens — verify JSON key name is correct
  • test_chat_request_uses_max_tokens_default — verify default key name
  • Schema merge tests for max_tokens_param propagation
  • YAML roundtrip test for LlmSection with max_tokens_param

All 621 tests pass. cargo fmt --all -- --check clean.

LLM API providers use different parameter names for max output tokens:
- OpenAI (legacy): max_tokens
- OpenAI (new): max_completion_tokens
- Anthropic: max_tokens
- Google Gemini/Vertex: max_output_tokens

cora-cli previously hardcoded 'max_tokens' everywhere, causing HTTP 400 errors
('Unsupported parameter: max_output_tokens') when providers don't accept it.

Changes:
- Add max_tokens_param to LlmSection config schema (supports 'auto' for
  provider-based detection, or explicit override)
- Add resolve_max_tokens_param() in loader.rs for auto-detection logic
- Replace hardcoded 'max_tokens' JSON key with dynamic key in both
  streaming and non-streaming LLM request paths
- Add max_tokens_param to LLMConfig struct for runtime use
- Display max_tokens_param in 'cora config' output

Auto-detection maps:
- gemini/google/vertex -> max_output_tokens
- All other providers -> max_tokens

Co-authored-by: Hermes Agent <hermes@nousresearch.com>
@ajianaz ajianaz merged commit 6793fc3 into develop Jul 7, 2026
9 of 10 checks passed
@ajianaz ajianaz deleted the fix/max-tokens-param-conditional branch July 7, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant