openai route: intermittent non-JSON output / schema failures on qwen3-vl-plus (double-closed JSON, legend region type)

[abyss-stars]

Environment

• modlens 3.12.1 (also reproduced on 3.11.0), run via the skill launcher (npx --yes --package @liustack/modlens@3.12.1 modlens)
• provider: openai, model: qwen3-vl-plus, baseUrl: DashScope (Alibaba Cloud Model Studio) OpenAI-compatible endpoint
• image: matplotlib Spearman correlation heatmap, 14×14 matrix, panel label a, color bar on the right
• Windows 11, node 25

Symptom

Intermittent failures on the openai route (roughly 1 in 2–3 runs), always with finish_reason=stop:

1. OpenAI-compatible API returned non-JSON output: {"summary":"..."}
2. OpenAI-compatible API returned JSON that does not match the vision schema (missing: layout.regions[3].type). Got: {"summary":...,"layout":{"regions":[...{"type":"legend",...}]...

Root cause 1 — the JSON object closes itself twice

I captured the raw response body through a local forward proxy. A failing generation looks like this (abbreviated):

{"summary":"...","ocr":{...},"layout":{...},"semantics":{"scene":"...","entities":[...],"relations":[...]}},"visual":{...},"uncertainty":[]}"

Note the extra } right after relations: the model closes the top-level object early, then keeps emitting ,"visual":{...},"uncertainty":[]} plus a stray trailing ". JSON.parse fails with "Unexpected non-whitespace character after JSON", and the old first-{-to-last-} brace slice fails too because of the early close. The error surfaces as the opaque non-JSON output, indistinguishable from a truncation (finish_reason=length).

The only parseable prefix is the partial object (keys: summary, ocr, layout, semantics), which then fails the schema check with missing: visual, uncertainty.

Root cause 2 — the schema enum rejects legend

The model labels the color-bar region "type":"legend"; the layout.regions[].type enum is title|subtitle|paragraph|list|table|chart|form|code|image|icon|other and rejects it, failing the whole read. legend is a normal region type for charts with color bars / map keys, and qwen3-vl-plus uses it consistently.

Suggested fixes

1. String-aware tolerant extraction: track brace depth outside string literals (escapes included) and keep the longest well-formed object. A stray close then yields the recoverable prefix and the schema check names the missing fields instead of "non-JSON output".
2. Report finish_reason and the content tail in parse-failure errors; give a dedicated message for finish_reason=length suggesting max_tokens in openai.extraBody.
3. Add legend to the layout.regions[].type enum (schema.ts, the JSON_TEMPLATE_INSTRUCTION in prompt.ts, dsh/vision-schema.json, docs/output-schema.md).
4. Document the gateway-side mitigation that removes the whole failure class at the source:
   modlens config set openai.extraBody '{"response_format":{"type":"json_object"}}'
   Verified 3/3 clean against DashScope qwen3-vl-plus: constrained decoding cannot emit the stray close or the trailing fragment.

Repro

modlens -i <heatmap.png> — run 2–4 times to hit the failure (generation is non-deterministic). Happy to attach a redacted failing raw response body if useful.

Note

A local patch implementing fixes 1–3 (with tests) exists; happy to turn it into a PR if that helps.

[abyss-stars]

Verified against the latest release (3.16.6) — one of the two issues is fixed, the other is only mitigated:

Fixed upstream: the legend schema rejection. layout.regions[].type is no longer an enum ("Deliberately not an enum... any other short label is fine"), so legend (and any other region kind the model chooses) passes. Confirmed live: reads that previously failed with missing: layout.regions[3].type now succeed. Thanks!

Not fixed: the double-closed JSON parse failure. src/util/json.ts extractJson is unchanged (still first-{-to-last-} brace slice). With response_format stripped (i.e. --extra-body '{}' and no structuredOutput), 3 of 5 runs against the same image failed again with OpenAI-compatible API returned non-JSON output — the same stray } + ,"visual":{...} + trailing " pattern. The route still depends on the gateway's JSON mode to be reliable.

The new structuredOutput knob works on DashScope, and is stronger than plain json_object. modlens config set openai.structuredOutput true sends response_format: {"type":"json_schema","json_schema":{...vision schema, strict:true}}; DashScope qwen3-vl-plus accepted it (no 400) and 2/2 runs returned schema-complete results, double-close impossible by construction. Worth documenting in configure.md as the recommended fix for flaky OpenAI-compatible gateways.

Suggestion: the string-aware tolerant extraction (keep the longest well-formed prefix, report finish_reason + tail on failure) from the issue body is still worth adopting — it makes the route survive gateways/models that cannot do response_format (and there are such gateways; e.g. some proxy/aggregator backends 400 on json_schema).

[liustack]

Both halves are fixed now, the second one in 3.17.1.

The double-closed JSON. Extraction sliced from the first { to the last }, which cannot survive an object that closes early and keeps writing: the slice swallowed the trailing fragment and failed, and the read died as non-JSON output, indistinguishable from a truncation exactly as you said. It now walks the braces with string literals and escapes accounted for and keeps the largest balanced span that parses, so your case comes back as the recoverable prefix and the schema check names visual and uncertainty instead of the parse dying. The old slice still runs first, so nothing it already read correctly changes.

The error now says which failure it was. finish_reason=length points at max_tokens, a stop points at openai.structuredOutput, and anything else (content_filter, a gateway's own reason) is reported as the gateway ending it early rather than described as a normal ending. Errors about how output ended now show the end of the output rather than the opening, since that is where the damage is.

One thing your report led to that was not in it. Showing the tail widened what an error can carry, and review caught that the HTTP error body was redacted against the API key only, not the configured baseUrl. A gateway writes its own response and can echo back what it received, so a private endpoint could reach a terminal, a CI log, or a failover warning. Every quotation of model or gateway text is now redacted against both, and redacted before it is clipped: a clip landing inside a hostname leaves nothing for an exact match to find. That was a live leak on the path your issue is about.

The legend enum was already gone, as you confirmed. And you were right that structuredOutput is the stronger answer where the gateway supports it: the tolerant extraction is for the gateways that answer 400 to json_schema, which is exactly the case you raised.

We do not take pull requests, so nothing was merged from your local patch, but the raw response body you captured through a proxy is what made this diagnosable, and coming back to verify which half was still broken on 3.16.6 is the reason the second half got fixed at all. Thank you.