Skip to content

feat: pass through all backend-supported sampling/serving parameters #151

Description

@ZuyiZhou

Problem

Raven forwards only a fixed subset of sampling parameters to the serving backend. Anything outside that subset is silently dropped, even when the backend supports it. Two incidents where this bit us:

  1. repetition_penalty — sglang supports it, but there is no way to get it through Raven's model config. We had to interpose a local proxy shim on the request path just to set one sampling knob.
  2. enable_thinking: false (Qwen3-style chat-template switch, normally passed via extra_body / chat_template_kwargs) — unreachable through Raven config; required patching the package.

Both are legitimate, backend-documented parameters; needing a fork for each new knob doesn't scale.

Proposal

A passthrough dict on the model/backend config, forwarded verbatim into the request payload (merged into the OpenAI-compatible body / extra_body):

model:
  sampling_extra:            # merged into sampling params
    repetition_penalty: 1.05
  request_extra:             # merged into extra_body
    chat_template_kwargs: { enable_thinking: false }

Exact shape is bikesheddable — the ask is: anything the backend accepts should be reachable from config without a fork.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions