Skip to content

Fix nightly model smoke: use json_object+schema response_format#441

Merged
charneykaye merged 2 commits into
mainfrom
fix/nightly-smoke-json-object-response-format
Jul 22, 2026
Merged

Fix nightly model smoke: use json_object+schema response_format#441
charneykaye merged 2 commits into
mainfrom
fix/nightly-smoke-json-object-response-format

Conversation

@charneykaye

@charneykaye charneykaye commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Problem

The Model smoke nightly workflow has failed on every run since it was added (2026-07-10). Each run fails with HTTP Error 500 when the smoke submits a player turn (example run). The static /start (NPC opening) succeeds because it never calls the model; the first real model call — /turn — 500s.

Root cause (the crash)

The llama_cpp adapter requests structured output using the OpenAI
{"type": "json_schema", "json_schema": {...}} form. The bundled llama.cpp
llama-server
(what production ships) accepts that, but the smoke uses
llama-cpp-python's OpenAI server as a stand-in, whose request model only
accepts type "text"/"json_object". It rejects the unknown "json_schema"
type with a 422, and the adapter turns any non-200 into a raised
ConnectionError → convsim-core returns 500 → the smoke's turn POST fails.

Fix

Send llama.cpp's original {"type": "json_object", "schema": {...}} form. Both
servers accept it, and it's actually more reliable than the json_schema type
on llama.cpp's own /v1/chat/completions (ggml-org/llama.cpp#11988). Adapter
unit test updated to assert the new shape.

Second issue the crash was masking (latency budget)

With the turn no longer crashing, the smoke measured full-turn latency for the
first time: a stable ~116 s on the 2-vCPU CPU-only runner (117.0 s / 115.3 s
over two runs), ~11.6× the 10 s documented budget. The guessed
--ci-hardware-factor 6 (72 s ceiling) was never validated because every prior
run crashed before timing a turn.

Fix

Calibrate --ci-hardware-factor 6 → 20 (200 s CI budget; 240 s ceiling with
the script's 1.20 tolerance) — ~2× headroom over the observed ~116 s so normal
runner variance does not flap the nightly, while a genuine >2× regression still
fails. This models CI-hardware slowness only; the product's 10 s target-hardware
SLO is unchanged.

Diagnosability

The smoke drained convsim-core's stderr to nothing, so the server-side
traceback never reached the CI logs — the client only ever saw HTTP Error 500.
It now retains a bounded stderr tail per child process and prints it on failure.

Verification

  • services/convsim-core tests pass locally (test_llama_cpp_runtime,
    test_turn_pipeline, test_debrief_engine, test_scripted_runtime,
    test_ollama_runtime — 250 passed).
  • Nightly workflow dispatched against this branch: turn succeeds, and with the
    calibrated factor the job is green (run 29902994858 → PASS). See the table in the PR comment.

🤖 Generated with Claude Code

charneykaye and others added 2 commits July 22, 2026 03:59
The llama_cpp adapter sent structured-output requests as the OpenAI
{"type": "json_schema", "json_schema": {...}} form. The bundled llama.cpp
llama-server accepts that, but llama-cpp-python's OpenAI server — which the
nightly smoke uses as a stand-in — only accepts type "text"/"json_object"
and rejects an unknown type with a 422. The adapter turns any non-200 into a
raised ConnectionError, so every player turn 500'd and the nightly smoke has
failed on every run since it was added (turn endpoint, not the static /start).

Switch to llama.cpp's original {"type": "json_object", "schema": {...}} form,
which both servers accept (and which is more reliable than the json_schema
type on llama.cpp's own /v1/chat/completions — ggml-org/llama.cpp#11988).

Also make the smoke self-diagnosing: it drained convsim-core's stderr to
nothing, so the server-side traceback never reached the CI logs (the client
only saw "HTTP Error 500"). Retain a bounded stderr tail per child process and
print it on failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the response_format 500 fixed, the smoke measured full-turn latency for
the first time: ~116 s on the 2-vCPU CPU-only runner (117.0 s and 115.3 s over
two consecutive runs), ~11.6x the 10 s documented budget. The prior 6x factor
(60 s CI budget) was never validated because every run crashed before timing a
turn, so the nightly failed on latency even after the crash fix.

Set the factor to 20 (200 s CI budget; 240 s ceiling with the 1.20 tolerance),
~2x headroom over the observed ~116 s so normal runner variance does not flap
while a genuine >2x regression still fails. Models CI-hardware slowness only;
the product's 10 s target-hardware SLO is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@charneykaye

Copy link
Copy Markdown
Contributor Author

Confirmed green end-to-end ✅

Dispatched Model smoke nightly against this branch three times:

Run full_response_ms Verdict
29902289670 117.0 s turn OK, latency FAIL (old 72 s ceiling)
29902684821 115.3 s turn OK, latency FAIL (old 72 s ceiling)
29902994858 114.5 s PASS (240 s ceiling) ✅

The first two runs prove the response_format fix resolved the HTTP 500 — the player turn now completes and the NPC returns valid grammar-constrained content. They also revealed a stable ~116 s per-turn latency on the 2-vCPU CPU-only runner (~11.6× the 10 s budget), which the guessed factor (72 s ceiling) never accounted for. The third run, with the factor calibrated to 20 (240 s ceiling, ~2× headroom), passes green.

@charneykaye
charneykaye merged commit e32dd1a into main Jul 22, 2026
20 checks passed
@charneykaye
charneykaye deleted the fix/nightly-smoke-json-object-response-format branch July 22, 2026 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant