Fix nightly model smoke: use json_object+schema response_format#441
Merged
Conversation
The llama_cpp adapter sent structured-output requests as the OpenAI
{"type": "json_schema", "json_schema": {...}} form. The bundled llama.cpp
llama-server accepts that, but llama-cpp-python's OpenAI server — which the
nightly smoke uses as a stand-in — only accepts type "text"/"json_object"
and rejects an unknown type with a 422. The adapter turns any non-200 into a
raised ConnectionError, so every player turn 500'd and the nightly smoke has
failed on every run since it was added (turn endpoint, not the static /start).
Switch to llama.cpp's original {"type": "json_object", "schema": {...}} form,
which both servers accept (and which is more reliable than the json_schema
type on llama.cpp's own /v1/chat/completions — ggml-org/llama.cpp#11988).
Also make the smoke self-diagnosing: it drained convsim-core's stderr to
nothing, so the server-side traceback never reached the CI logs (the client
only saw "HTTP Error 500"). Retain a bounded stderr tail per child process and
print it on failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the response_format 500 fixed, the smoke measured full-turn latency for the first time: ~116 s on the 2-vCPU CPU-only runner (117.0 s and 115.3 s over two consecutive runs), ~11.6x the 10 s documented budget. The prior 6x factor (60 s CI budget) was never validated because every run crashed before timing a turn, so the nightly failed on latency even after the crash fix. Set the factor to 20 (200 s CI budget; 240 s ceiling with the 1.20 tolerance), ~2x headroom over the observed ~116 s so normal runner variance does not flap while a genuine >2x regression still fails. Models CI-hardware slowness only; the product's 10 s target-hardware SLO is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
Confirmed green end-to-end ✅Dispatched Model smoke nightly against this branch three times:
The first two runs prove the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Model smoke nightly workflow has failed on every run since it was added (2026-07-10). Each run fails with
HTTP Error 500when the smoke submits a player turn (example run). The static/start(NPC opening) succeeds because it never calls the model; the first real model call —/turn— 500s.Root cause (the crash)
The
llama_cppadapter requests structured output using the OpenAI{"type": "json_schema", "json_schema": {...}}form. The bundled llama.cppllama-server(what production ships) accepts that, but the smoke usesllama-cpp-python's OpenAI server as a stand-in, whose request model only
accepts
type"text"/"json_object". It rejects the unknown"json_schema"type with a 422, and the adapter turns any non-200 into a raised
ConnectionError→ convsim-core returns 500 → the smoke's turn POST fails.Fix
Send llama.cpp's original
{"type": "json_object", "schema": {...}}form. Bothservers accept it, and it's actually more reliable than the
json_schematypeon llama.cpp's own
/v1/chat/completions(ggml-org/llama.cpp#11988). Adapterunit test updated to assert the new shape.
Second issue the crash was masking (latency budget)
With the turn no longer crashing, the smoke measured full-turn latency for the
first time: a stable ~116 s on the 2-vCPU CPU-only runner (117.0 s / 115.3 s
over two runs), ~11.6× the 10 s documented budget. The guessed
--ci-hardware-factor 6(72 s ceiling) was never validated because every priorrun crashed before timing a turn.
Fix
Calibrate
--ci-hardware-factor6 → 20(200 s CI budget; 240 s ceiling withthe script's 1.20 tolerance) — ~2× headroom over the observed ~116 s so normal
runner variance does not flap the nightly, while a genuine >2× regression still
fails. This models CI-hardware slowness only; the product's 10 s target-hardware
SLO is unchanged.
Diagnosability
The smoke drained convsim-core's stderr to nothing, so the server-side
traceback never reached the CI logs — the client only ever saw
HTTP Error 500.It now retains a bounded stderr tail per child process and prints it on failure.
Verification
services/convsim-coretests pass locally (test_llama_cpp_runtime,test_turn_pipeline,test_debrief_engine,test_scripted_runtime,test_ollama_runtime— 250 passed).calibrated factor the job is green (run 29902994858 → PASS). See the table in the PR comment.
🤖 Generated with Claude Code