Description
The OpenAI-compatible /v1/chat/completions endpoint accepts response_format but silently ignores it. There is no error, no warning, and no effect on output — the model returns plain text regardless of the value passed.
Reproduction
Request:
curl -s http://localhost:52625/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gemma4-it:e4b",
"messages": [
{"role": "system", "content": "You are a helpful assistant that always responds in valid JSON format."},
{"role": "user", "content": "List 3 programming languages with their type system and year created. Return as a JSON object with a \"languages\" array."}
],
"max_tokens": 300,
"response_format": {"type": "json_object"}
}'
Actual response (content):
{
"languages":
{
"name": "Java",
"type": "static",
"year": 1995
},
...
}
This JSON object contains a list of...
Expected:
Raw valid JSON, parseable without post-processing (matching OpenAI API behavior when response_format: {"type": "json_object"} is set).
Tests performed
Tests performed :
| response_format value | Result |
| {"type": "json_object"} | ❌ Ignored — markdown-wrapped output|
| {"type": "text"} |❌ Ignored — same behavior as default|
Not set (default) | Same output as above
Passing an unrecognized value also produces no error — the parameter appears to be completely ignored server-side.
Expected behavior
- response_format: {"type": "json_object"} should force the model to output valid JSON (no markdown fences, no conversational wrapper).
- Invalid/unrecognized response_format values should return a 400 error rather than being silently ignored.
- (Optional) Support {"type": "json_schema", "json_schema": {...}} for schema-constrained output, matching the OpenAI Structured Outputs API (https://platform.openai.com/docs/guides/structured-outputs).
Why this matters
- OpenAI API compatibility: response_format is a core parameter of the OpenAI Chat Completions API. Clients (OpenAI SDK, LangChain, etc.) rely on it for structured parsing.
- Downstream usage: Any pipeline that expects parseable JSON from the response (tool calling, RAG, data extraction) currently has to implement ad-hoc post-processing (strip code fences, regex-extract JSON blocks), which is fragile and model-dependent.
- Competitors: llama.cpp (GBNF grammars), Ollama ("format": "json"), vLLM, and LM Studio all support this.
Environment
- FLM version: v0.9.42
- Model tested: gemma4-it:e4b
- OS: Linux
- Endpoint: POST /v1/chat/completions
Description
The OpenAI-compatible /v1/chat/completions endpoint accepts response_format but silently ignores it. There is no error, no warning, and no effect on output — the model returns plain text regardless of the value passed.
Reproduction
Request:
Expected:
Raw valid JSON, parseable without post-processing (matching OpenAI API behavior when response_format: {"type": "json_object"} is set).
Tests performed
Tests performed :
| response_format value | Result |
| {"type": "json_object"} | ❌ Ignored — markdown-wrapped output|
| {"type": "text"} |❌ Ignored — same behavior as default|
Not set (default) | Same output as above
Passing an unrecognized value also produces no error — the parameter appears to be completely ignored server-side.
Expected behavior
Why this matters
Environment