test(tutorials): re-enable orchestrate-a-sonnet-with-workflows - #893
Merged
Conversation
Removes the last temporary ignore entry. Its causes were fixed in #881 — the unpulled `llama3.2` model, the annotation bound to the wrong command, the missing `MAX_COMPLETION_TOKENS`, `revise` re-entering `create_text` for ten generations instead of six, and no `on_failure` on the dispatch states. What the entry was waiting on was a green run, and CI on this PR is that run. Verified statically against the merged file first, since the annotation placement is the bug class that broke it before: - all three annotations (`# → retry 90`, `# → retry 40`, `# → 400`) sit on their own line immediately before the command they apply to, which is the only placement tests/tutorials-tests.sh honors; - `STATES` and `TRANSITIONS` both parse as JSON once `'"$AGENT_ID"'` is substituted; - single-quote parity is even in all 10 CLI blocks, so no comment desyncs the runner's line accumulator; - all 10 tab groups carry cli/sdk/curl; zero `SOAT_URL`; zero `llama3.2`; - `on_failure` present on every dispatch state; `revise` targets `stanza_4`, so the rework pass costs one generation rather than five. The ignore list is now three entries, all of them permanent: `index` (a landing page with no commands) and two tutorials that need real third-party or cloud credentials CI should not hold. Timing is the remaining risk, not correctness. The forward chain is five sequential generations against `qwen2.5:0.5b` on a CPU-only runner, capped at 256 output tokens, with a 90-attempt (~90s) retry budget; the revise pass is one generation with 40. If it times out rather than failing, raising those budgets is the fix, not re-ignoring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjeWpxDqRDtkSv1mCDspf9
Deploy Outputs
|
The first CI run of the re-enabled tutorial timed out rather than failing: the card reached `review` normally, but 90 attempts ran out before it got there. Server logs show the whole chain completing. 10:07:11 start 10:07:44 to_stanza_1 (+32s) 10:08:55 to_stanza_2 (+72s) 10:09:32 to_stanza_3 (+36s) 10:09:54 to_stanza_4 (+22s) 10:10:10 to_review (+16s) --------------------------------- forward chain: 179s So the workflow logic and every #881 fix are sound; only the budget was wrong. It was wrong because I sized it by extrapolating the ~9s/completion implied by the smoke-suite profiling in tests.md. That number does not transfer: these prompts carry the accumulated poem forward, so input grows each hop, and per-generation cost here ranged 16-72s. A poll attempt also costs its 1s sleep plus a CLI round trip, so 90 attempts bought roughly 135s of wall-clock against a 179s chain. Forward chain 90 -> 240 attempts (~2x the observed total, so ordinary variance does not fail the run) and the revise pass 40 -> 120. The SDK tab's `waitForState` loop is raised to match. A budget is a ceiling, not a cost: the loop exits as soon as the card lands. The measured timings are now recorded in the step's prose, so the next person sizing this has data instead of an extrapolation. Worth weighing separately: this tutorial contributes about three minutes of Ollama time to an already Ollama-bound job. Cutting the chain from four stanzas to two would roughly halve that, at the cost of the Shakespearean structure the tutorial is built around. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjeWpxDqRDtkSv1mCDspf9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the last temporary entry from
tests/.tutorialsignore. Its causes were fixed in #881; what the entry was waiting on was a green run, andTutorials Testson this PR is that run.Recap of what #881 fixed:
llama3.2, which this stack never pulls# → 400sat after its command, so it bound to the next oneMAX_COMPLETION_TOKENSon the tutorials stackrevisere-enteredcreate_texton_failureon dispatch statesVerified before re-enabling
Checked statically against the merged file, focusing on the annotation placement that broke it before:
# → retry 90,# → retry 40,# → 400) sit on their own line immediately before their command — the only placementtests/tutorials-tests.shhonors. Confirmed by reading the lines, not by assuming.STATESandTRANSITIONSboth parse as JSON once'"$AGENT_ID"'is substituted. (A first pass reportedSTATESinvalid; that was my substitution replacing an already-quoted interpolation and producing""X""— an artifact, not a defect.)SOAT_URL; zerollama3.2;on_failureon every dispatch state;revise→stanza_4.Ignore list after this change
Three entries, all permanent:
indexconnect-third-party-llmscall-aws-and-gcp-apis-from-an-agentThat takes the suite from 24 tutorials to 25 of 28 files running in CI.
Risk
Timing, not correctness. The forward chain is five sequential generations against
qwen2.5:0.5bon a CPU-only runner, now capped at 256 output tokens, with a 90-attempt (~90s) retry budget; the revise pass is one generation with 40. At the ~9s/completion the smoke profiling suggests, five generations land around 45s — inside budget, but not by a wide margin.If it times out rather than failing outright, the fix is raising those retry budgets, not re-ignoring the tutorial. A genuine failure (a stalled card, a dispatch error) would be a different signal and worth reading properly.
As with #890, one green run is a first-run signal, not a stability signal.
Related: #881, #890
🤖 Generated with Claude Code
https://claude.ai/code/session_01AjeWpxDqRDtkSv1mCDspf9
Generated by Claude Code