fix(tests): make the sonnet-workflows tutorial CI-viable

[arantespp]

Summary

orchestrate-a-sonnet-with-workflows sits in .tutorialsignore on the grounds that its on_enter agent dispatch is inherently non-deterministic, "same class as review-panel-discussion." That diagnosis is wrong on both counts, and two of the real causes are outright bugs rather than flakiness.

It is not the same class. review-panel-discussion is ignored because qwen2.5:0.5b cannot reliably emit the structured JSON its agent nodes declare via output_schema — a model-capability problem. This tutorial declares no output_schema anywhere, and every on_complete is {"when": true}, so the card advances on any completion regardless of content. Nothing here depends on model quality — only on completion.

This PR fixes the causes. It deliberately does not remove the .tutorialsignore entry; see Why the ignore entry stays below.

What was actually broken

1. The AI provider pinned llama3.2, which the CI stack never pulls. docker-compose.tutorials.yml pulls qwen2.5:0.5b only, so every generation in this tutorial would fail outright — a deterministic failure that the "flaky LLM" framing hid entirely. Now qwen2.5:0.5b, per .claude/rules/tutorials.md.

2. The # → 400 annotation was placed where it silently binds to the wrong command. It sat on its own line after the command. In tutorials-tests.sh, a command line with balanced quotes is flushed the moment it is read, so CURRENT_CMD is already empty and the hint becomes NEXT_ANNOTATION — it bound to update-task. Net effect: the expected guard rejection was unannotated and failed the run, while update-task was wrongly expected to fail. Moved before the command.

3. The tutorials stack never received MAX_COMPLETION_TOKENS. Both stacks proxy Ollama through the same ollamaToolChoiceProxy.mjs, which implements the cap generically ("Unset = no cap") and notes that agents carry no max_tokens of their own. The smoke stack sets 256; tutorials set nothing. Uncapped output is what made the smoke job swing 219s → 406s between identical runs. Capping it here bounds every tutorial's wall-clock, not just this one.

4. revise pointed at create_text, re-running all five generations on the rework pass — ten per tutorial run. It now targets stanza_4 for a fresh closing couplet: one generation instead of five, with the backward-move lesson unchanged. The prose makes the modelling choice explicit, since re-entering at the step that needs redoing is the actual point.

5. No state declared on_failure, so a generation error parked the card with no route out — a hang rather than a failure. Every dispatch state now falls back to a new abandon_to_review transition.

6. Two hand-rolled for … sleep 2 poll loops are replaced with # → retry N over a jq -e assertion — what that annotation documents itself as replacing.

It also brings the file up to authoring rules it predates: curl tabs used SOAT_URL instead of SOAT_BASE_URL, there were no SDK tabs at all (against the three-tab rule), and the board query's curl tab read .[] where the endpoint returns {data: [...]}.

Root cause of #2 — a docs bug that generates this mistake

.claude/rules/tests.md claimed "a trailing inline annotation applies to the command it sits on". That is false. The extractor only matches # at the start of a line, so an inline # → … reaches the shell as an ordinary comment and the annotation is lost. The same passage said an annotation on its own line applies to the next command — true, but it reads as though placing one after a command works.

Both are corrected, with all three placements shown explicitly, and the stale comment in tutorials-tests.sh claiming inline annotations are "preserved as a synthetic marker line" now describes what the awk actually does. Behavior is pinned by tests/harness/tutorialsRetryAnnotation.test.mjs, where every case places the annotation on the preceding line.

This matters beyond one file: #878 inherited the same bug from this documentation (a trailing # → expect-fail). Fixing that is a one-line follow-up on that branch.

Why the ignore entry stays

Un-ignoring makes a flake block every PR, and the Docker daemon is unavailable in the environment this was written in, so the tutorial has not been executed end to end. Removing the entry belongs in a follow-up with green runs behind it — TUTORIAL_ID=orchestrate-a-sonnet-with-workflows docker compose -f tests/docker-compose.tutorials.yml up …, a few times.

Fix 3 is worth landing on its own merits regardless, since it bounds runtime for every tutorial in the suite.

Verification

• node scripts/docs-lint.mjs — OK, 460 files, no violations
• node --test tests/harness/tutorialsRetryAnnotation.test.mjs — 6/6 pass (the runner change is comment-only, but this pins the annotation contract the docs now describe)
• pnpm --filter @soat/website build — exit 0, no broken links or anchors
• docker-compose.tutorials.yml parses; proxy env resolves to {PORT, OLLAMA_UPSTREAM_URL, TOOL_CHOICE_TOOLS, MAX_COMPLETION_TOKENS: '256'}
• Static checks on the extracted CLI blocks: 10 blocks, single-quote parity even in every one, both multi-line JSON payloads parse after shell interpolation, all 10 <Tabs> groups now carry cli/sdk/curl, zero SOAT_URL, zero llama3.2
• on_failure verified against workflowsValidation.ts:197 — it references a transition name, not a state, which is why abandon_to_review had to be added rather than pointing at review directly

Not verified: the tutorial has not run against a live server (no Docker daemon available). That is exactly why the ignore entry stays in this PR.

Related: #878

🤖 Generated with Claude Code

https://claude.ai/code/session_01AjeWpxDqRDtkSv1mCDspf9

---

Generated by Claude Code

[github-actions]

Deploy Outputs

Package	Stack	Output Key	Output Value
@soat/website	SoatWebsite-claude-soat-graphed-system-z169oo-tutorials-determinism	BucketWebsiteURL	http://soatwebsite-claude-soat-graphed-syste-staticbucket-1kp3zeh0p9wf.s3-website-us-east-1.amazonaws.com