From b52cf749f53c1985e5772645451aa1337d90d5bd Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 8 Aug 2026 10:02:30 +0000 Subject: [PATCH 1/2] test(tutorials): re-enable orchestrate-a-sonnet-with-workflows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removes the last temporary ignore entry. Its causes were fixed in #881 — the unpulled `llama3.2` model, the annotation bound to the wrong command, the missing `MAX_COMPLETION_TOKENS`, `revise` re-entering `create_text` for ten generations instead of six, and no `on_failure` on the dispatch states. What the entry was waiting on was a green run, and CI on this PR is that run. Verified statically against the merged file first, since the annotation placement is the bug class that broke it before: - all three annotations (`# → retry 90`, `# → retry 40`, `# → 400`) sit on their own line immediately before the command they apply to, which is the only placement tests/tutorials-tests.sh honors; - `STATES` and `TRANSITIONS` both parse as JSON once `'"$AGENT_ID"'` is substituted; - single-quote parity is even in all 10 CLI blocks, so no comment desyncs the runner's line accumulator; - all 10 tab groups carry cli/sdk/curl; zero `SOAT_URL`; zero `llama3.2`; - `on_failure` present on every dispatch state; `revise` targets `stanza_4`, so the rework pass costs one generation rather than five. The ignore list is now three entries, all of them permanent: `index` (a landing page with no commands) and two tutorials that need real third-party or cloud credentials CI should not hold. Timing is the remaining risk, not correctness. The forward chain is five sequential generations against `qwen2.5:0.5b` on a CPU-only runner, capped at 256 output tokens, with a 90-attempt (~90s) retry budget; the revise pass is one generation with 40. If it times out rather than failing, raising those budgets is the fix, not re-ignoring. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01AjeWpxDqRDtkSv1mCDspf9 --- tests/.tutorialsignore | 20 -------------------- 1 file changed, 20 deletions(-) diff --git a/tests/.tutorialsignore b/tests/.tutorialsignore index c1218691..811ddce4 100644 --- a/tests/.tutorialsignore +++ b/tests/.tutorialsignore @@ -8,26 +8,6 @@ index connect-third-party-llms -# orchestrate-a-sonnet-with-workflows is disabled pending evidence, not pending a -# fix. Its original rationale — inherent `on_enter` dispatch nondeterminism, -# "same class as review-panel-discussion" — was wrong on both counts and is -# corrected here. The tutorial declares no `output_schema` anywhere and every -# `on_complete` is `{"when": true}`, so the card advances on any completion -# regardless of content; nothing in it depends on model quality. The actual -# causes were fixed in #881: -# - the AI provider pinned `llama3.2`, which this stack never pulls (it pulls -# `qwen2.5:0.5b` only), so every generation failed outright; -# - the `# → 400` annotation sat after its command, so it bound to the next -# one, leaving the expected guard rejection unannotated; -# - this stack had no `MAX_COMPLETION_TOKENS`, so output was unbounded; -# - `revise` re-entered `create_text`, running ten generations per run instead -# of six; -# - no state declared `on_failure`, so a failed generation stalled the card. -# What is still missing is a green run: nothing has executed this tutorial end to -# end since those fixes landed. Remove this entry once -# `TUTORIAL_ID=orchestrate-a-sonnet-with-workflows` passes locally a few times in -# a row. -orchestrate-a-sonnet-with-workflows # call-aws-and-gcp-apis-from-an-agent creates tools whose `execute.auth` signs # requests with real AWS credentials or mints a GCP service account token. The From 588ae03a1933705030c3d2d0e831ffe2a8733d10 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 8 Aug 2026 10:14:31 +0000 Subject: [PATCH 2/2] fix(website): size the sonnet-workflow retry budgets from measurement The first CI run of the re-enabled tutorial timed out rather than failing: the card reached `review` normally, but 90 attempts ran out before it got there. Server logs show the whole chain completing. 10:07:11 start 10:07:44 to_stanza_1 (+32s) 10:08:55 to_stanza_2 (+72s) 10:09:32 to_stanza_3 (+36s) 10:09:54 to_stanza_4 (+22s) 10:10:10 to_review (+16s) --------------------------------- forward chain: 179s So the workflow logic and every #881 fix are sound; only the budget was wrong. It was wrong because I sized it by extrapolating the ~9s/completion implied by the smoke-suite profiling in tests.md. That number does not transfer: these prompts carry the accumulated poem forward, so input grows each hop, and per-generation cost here ranged 16-72s. A poll attempt also costs its 1s sleep plus a CLI round trip, so 90 attempts bought roughly 135s of wall-clock against a 179s chain. Forward chain 90 -> 240 attempts (~2x the observed total, so ordinary variance does not fail the run) and the revise pass 40 -> 120. The SDK tab's `waitForState` loop is raised to match. A budget is a ceiling, not a cost: the loop exits as soon as the card lands. The measured timings are now recorded in the step's prose, so the next person sizing this has data instead of an extrapolation. Worth weighing separately: this tutorial contributes about three minutes of Ollama time to an already Ollama-bound job. Cutting the chain from four stanzas to two would roughly halve that, at the cost of the Shakespearean structure the tutorial is built around. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01AjeWpxDqRDtkSv1mCDspf9 --- .../orchestrate-a-sonnet-with-workflows.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/packages/website/docs/tutorials/orchestrate-a-sonnet-with-workflows.md b/packages/website/docs/tutorials/orchestrate-a-sonnet-with-workflows.md index 1bb51ae5..1bed429b 100644 --- a/packages/website/docs/tutorials/orchestrate-a-sonnet-with-workflows.md +++ b/packages/website/docs/tutorials/orchestrate-a-sonnet-with-workflows.md @@ -535,13 +535,20 @@ retry: `# → retry N` re-runs the command until it exits `0`, up to `N` attempt second apart. `jq -e` supplies the exit code — non-zero until `.state` is actually `review`. +Size `N` from measurement, not optimism. On the CI sandbox (`qwen2.5:0.5b`, CPU +only, output capped at 256 tokens) the five generations took **32s, 72s, 36s, +22s and 16s — 179s end to end**, and each poll attempt costs its one-second sleep +plus the CLI round trip. The budget below is roughly double the observed total so +ordinary variance does not fail the run; it is a ceiling, not a cost, since the +loop exits as soon as the card lands. + ```bash soat transition-task --task-id "$TASK_ID" --transition start | jq '{ state, automation_status }' -# → retry 90 +# → retry 240 soat get-task --task-id "$TASK_ID" | jq -e '.state == "review"' soat get-task --task-id "$TASK_ID" | jq '{ state, status, sonnet: .last_result.content }' @@ -557,7 +564,8 @@ await adminSoat.tasks.transitionTask({ }); const waitForState = async (target: string) => { - for (let attempt = 0; attempt < 90; attempt += 1) { + // ~4 minutes: the five sandbox generations measured 179s end to end. + for (let attempt = 0; attempt < 240; attempt += 1) { const { data: current } = await adminSoat.tasks.getTask({ path: { task_id: TASK_ID }, }); @@ -616,7 +624,7 @@ modelling decision, and it is the difference between a fast loop and a slow one. ```bash soat transition-task --task-id "$TASK_ID" --transition revise --note "tighten the closing couplet" | jq '{ state }' -# → retry 40 +# → retry 120 soat get-task --task-id "$TASK_ID" | jq -e '.state == "review"' ```