test(tutorials): correct a stale ignore rationale and re-enable the review panel - #890
Merged
Merged
Conversation
…eview panel
Two changes to tests/.tutorialsignore.
**orchestrate-a-sonnet-with-workflows** — the comment was wrong, and it was
wrong in a way that discouraged anyone from retrying the tutorial. It
claimed the tutorial was inherently nondeterministic because of `on_enter`
agent dispatch, "same class as review-panel-discussion". Neither half held:
the tutorial declares no `output_schema` anywhere and every `on_complete`
is `{"when": true}`, so the card advances on any completion regardless of
what the model produced. Nothing in it depended on model quality.
The real causes were mundane and are fixed on main (#881): the provider
pinned `llama3.2`, which this stack never pulls; the `# → 400` annotation
sat after its command so it bound to the next one; the stack had no
`MAX_COMPLETION_TOKENS`, leaving output unbounded; `revise` re-entered
`create_text` for ten generations per run instead of six; and no state
declared `on_failure`, so a failed generation stalled the card.
The entry stays because what is missing now is evidence, not a fix — the
tutorial has not been executed end to end since those landed. The comment
says so and names the command that would settle it.
**review-panel-discussion** — removed from the ignore list, so CI runs it
again. Its documented cause is unchanged and unaddressed: both agent nodes
declare an `output_schema` and `buildStructuredOutput` validates the model's
JSON with ajv, so a 0.5b model that emits a non-conforming object fails the
generation and the run. This re-enables it to find out whether that still
reproduces, rather than on the strength of a fix.
Static checks on the tutorial pass — 11 CLI blocks with even single-quote
parity, both JSON payloads valid, all three tabs on every block, the
correct `qwen2.5:0.5b` model, no `SOAT_URL`, no annotations — so a
structural failure of the kind that broke the workflows tutorial is ruled
out. The runtime question is not.
If it flakes, the revert is re-adding one line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjeWpxDqRDtkSv1mCDspf9
Deploy Outputs
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes to
tests/.tutorialsignore.orchestrate-a-sonnet-with-workflows— comment corrected, entry keptThe rationale was wrong, and wrong in a way that discouraged anyone from retrying the tutorial. It claimed inherent nondeterminism from
on_enteragent dispatch, "same class as review-panel-discussion." Neither half held: the tutorial declares nooutput_schemaanywhere and everyon_completeis{"when": true}, so the card advances on any completion regardless of content. Nothing in it depended on model quality.The real causes were mundane, and all are fixed on
main(#881):llama3.2, which this stack never pulls# → 400sat after its command, so it bound to the next oneMAX_COMPLETION_TOKENSrevisere-enteredcreate_texton_failureThe entry stays because what is missing now is evidence, not a fix — nothing has executed the tutorial end to end since those landed. The comment now says exactly that and names the command that would settle it (
TUTORIAL_ID=orchestrate-a-sonnet-with-workflows).review-panel-discussion— removed, so CI runs it againIts documented cause is unchanged and unaddressed: both agent nodes declare an
output_schema, andbuildStructuredOutput(outputSchema.ts) validates the model's JSON with ajv, so a 0.5b model emitting a non-conforming object fails the generation and the run. This re-enables it to find out whether that still reproduces — not on the strength of a fix.I looked for the provider-boundary fix
.claude/rules/tests.mdprefers over retrying a generation, and did not find one I could justify:ollamaToolChoiceProxy.mjsalready forwards bodies verbatim, so whatever the AI SDK sends forOutput.objectreaches Ollama untouched. Bridging structured output there would be speculative without being able to run the stack, so I did not invent it.What is and is not verified
Static checks on
review-panel-discussionpass, which rules out the bug classes that actually broke the workflows tutorial:ORCH_NODES,ORCH_EDGES) parse after shell interpolation<Tabs>groups carry cli/sdk/curlqwen2.5:0.5bmodel; zeroSOAT_URL; no annotations to misplaceThe runtime question is not verified. No Docker daemon is available in the environment this was written in, so the
Tutorials Testsjob on this PR is the experiment. Two possible outcomes:output_schema×qwen2.5:0.5bis still the blocker. Revert is re-adding one line; the follow-up would then be either dropping the schemas (which changes what the tutorial teaches) or implementing constrained decoding at the proxy.Note this is a first-run signal, not a stability signal — the original comment asked for repeated green runs before trusting it, which one PR run does not provide.
Ignore list after this change
indexconnect-third-party-llmsorchestrate-a-sonnet-with-workflowscall-aws-and-gcp-apis-from-an-agentOne gap I left alone to keep this focused:
connect-third-party-llmsis the only entry with no explanatory comment, and it doubles as the example in the file header, which reads confusingly.Related: #881
🤖 Generated with Claude Code
https://claude.ai/code/session_01AjeWpxDqRDtkSv1mCDspf9
Generated by Claude Code