feat: reviewer turn backstop (maxTurns) + §12a missing-verdict guard (2.5.14) - #613
feat: reviewer turn backstop (maxTurns) + §12a missing-verdict guard (2.5.14)#613jstrunk wants to merge 4 commits into
Conversation
…ays enforced Review feedback on PR awslabs#613: persona prose in core/agents/*.md should not carry harness-specific instructions, and telling the model "on Codex and Kiro no native cap exists; this is prose you hold yourself to" undermines the budget on exactly the harnesses without enforcement. - Both reviewers' '## Turn Budget' enforcement bullet is rewritten unconditionally: a HARD cap of 40 turns; when hit you are stopped, in the worst case with NO final message and NO chance to write anything - so plan for the worst case, protect the reserve, and write the review before the cap, never on the last turn. All harness names and per-harness cutoff semantics removed; the frontmatter/prose number-sync note stays (harness-free wording); per-persona voicing kept. - The per-harness enforcement matrix remains where it belongs: docs/reference/04-stage-protocol.md and docs/reference/agents/ README.md (maintainer/user-facing), unchanged. - opencode emit: alongside the maxTurns->steps frontmatter rename, the persona body's own `maxTurns: 40` citation is renamed to `steps: 40` so the shipped roster's prose matches the key the harness actually reads. - t243: persona pins now scope to the Turn Budget section, assert the worst-case-no-output semantics and number sync, and add a negative pin (t221-style): no harness names and no steps: key inside the core persona section. The opencode dist translation pins (steps: 40 present, no maxTurns leak, no other agent capped) are unchanged. - CHANGELOG 2.5.2: the persona bullet now describes the section as deliberately harness-neutral, with the enforcement matrix living in the docs. References upstream issue awslabs#609.
|
Reviewed at head 591f66a. Verdict: the design is sound and the external-harness claims all check out against primary sources. One real protocol gap on the revision path (finding 1 below) and the usual rebase/slot logistics (finding 2 below) before this can land; the rest are smaller. Findings ranked. Verified first (primary sources, not the PR text)
Findings1. P1 - the guard still accepts a stale READY on the Part 0 revision path. Sequence: a stage passes review READY at iteration 1; the human rejects at the gate; the builder revises a Two ways to close it: (a) the conductor removes or renames the stale section (e.g. 2. P1 - rebase + slot renumbering. The branch is CONFLICTING against v2, and both of its slots are now taken: v2 is at 2.5.2 (#576 merged, heading 3. P2 - the missing-verdict branch spends a full builder dispatch on a finding the builder cannot act on. "review did not complete within its turn budget" feeds the existing NOT-READY loop, whose first step re-invokes the lead alone to address the findings - but this finding is not an artifact defect; there is nothing for the builder to fix. Consider short-circuiting this specific finding straight to the reviewer re-invoke (still counting the iteration). Not blocking - the current shape is simple and bounded - but it wastes a whole subagent run per cutoff. 4. P2 - the Codex TOML persona points at frontmatter that does not exist on that surface.
5. P3 - the agent-authoring guide omits the new key.
6. P3 - verify-item: the kiro-ide .md frontmatter parser. The kiro-ide dist personas now carry an unknown frontmatter key on the surface the IDE actually parses for subagent tool grants (the frontmatterAdditions seam in the kiro-ide manifest). Almost certainly ignored as an unknown key, but the PR's schema research covered the agent-v1 JSON, not the IDE's .md frontmatter parser - one sentence of confirmation from your kiro-ide probing would close it. On the test (t243, to become t244)Good coverage shape: both sides of the number-sync contract, the worst-case-cutoff wording, the harness-name negative pin, the opencode translation (exactly one |
Field reviews ran 56-113 turns each; nothing bounded how much work a §12a reviewer did inside its read scope, and a reviewer stopped before writing its verdict left the conductor on an undefined branch. Three coordinated changes (upstream issue awslabs#609): - maxTurns: 40 in both reviewer agents' frontmatter (binding on Claude Code; ships inertly in the kiro/opencode/codex .md copies - the packager's projectTierFrontmatter rewrites only the tier: line, so the key passes through verbatim; not added to Kiro agent JSON or the Codex TOML headers). - '## Turn Budget' persona section in both reviewers, voiced per-persona: cutoff semantics (stopped mid-task, no warning, no final-message turn, caller receives no output), the ~15 read / ~5 validation / ~10 verify / final ~5 reserved-for-writing split, and the standing order that a verdict backed by fewer verified findings always beats no verdict. - §12a step 3 missing-verdict branch (additive after the dispatch-record deletion t221 pins): a primary artifact with no '## Review' section for THIS iteration reads as NOT-READY with the finding "review did not complete within its turn budget" and counts against reviewer_max_iterations; the existing loop and iterations-exhausted paths apply unchanged. Mirrored in all five harness SKILL.md reviewer bullets (kiro-ide stays dispatch-record-free). New pin test tests/unit/t248-reviewer-turn-budget.test.ts guards the frontmatter cap, the number sync with the persona prose, and the missing-verdict sentence on every surface (core + dist). Docs updated (04-stage-protocol, guide/06-agents, glossary, reference/agents README). Version 2.5.14 + CHANGELOG + README badge; dist regenerated via bun scripts/package.ts (--check green). References upstream issue awslabs#609.
…ss enforcement notes Research (2026-07-20, official docs + source): opencode has a native per-agent step cap - `steps` (positive integer, agent .md frontmatter or opencode.json; legacy `maxSteps` deprecated; shipped since v1.0.134). Its cutoff differs from Claude Code's: the runner forces a final TEXT-ONLY turn (tools disabled, "MAXIMUM STEPS REACHED"), so the agent returns a summary but cannot make the tool call that writes the '## Review' section - the §12a missing-verdict guard still matters there. Codex CLI has no turn/iteration cap key (nearest is features.rollout_budget.limit_tokens - token-based, session-tree-wide; verified against the config reference and a rust-v0.144.6 source grep). Kiro CLI/IDE agent-v1 is additionalProperties:false with no cap field (v3/KAS adds caps only to /goal --max and agent_crew loop_config - orchestration surfaces, not agent JSON). - harness/opencode/emit.ts (emitSubagentMd): rename a core `maxTurns:` frontmatter line to opencode's native `steps:` in the .opencode/ native roster (the only dir opencode reads) - same seam as the existing tier->model/variant projection. The authored core key stays maxTurns (single authored source). - Turn Budget persona sections (both reviewers): per-harness accurate enforcement sentence - hard-capped on Claude Code (stopped mid-task, no final message) AND opencode (final turn text-only: summary yes, review write no); prose-only on Codex and Kiro (no native cap exists). - Docs: 04-stage-protocol, guide/06-agents, glossary, and reference/agents README now state the per-harness enforcement matrix with one-line reasons for the Codex/Kiro absence. - t248: new pin - opencode dist reviewers carry `steps: 40` with no `maxTurns:` leak, and no other opencode agent gains a steps cap; cutoff-semantics pin loosened to per-persona voicing. - CHANGELOG 2.5.14 bullets updated to the researched matrix (version stays 2.5.14). References upstream issue awslabs#609.
…ays enforced Review feedback on PR awslabs#613: persona prose in core/agents/*.md should not carry harness-specific instructions, and telling the model "on Codex and Kiro no native cap exists; this is prose you hold yourself to" undermines the budget on exactly the harnesses without enforcement. - Both reviewers' '## Turn Budget' enforcement bullet is rewritten unconditionally: a HARD cap of 40 turns; when hit you are stopped, in the worst case with NO final message and NO chance to write anything - so plan for the worst case, protect the reserve, and write the review before the cap, never on the last turn. All harness names and per-harness cutoff semantics removed; the frontmatter/prose number-sync note stays (harness-free wording); per-persona voicing kept. - The per-harness enforcement matrix remains where it belongs: docs/reference/04-stage-protocol.md and docs/reference/agents/ README.md (maintainer/user-facing), unchanged. - opencode emit: alongside the maxTurns->steps frontmatter rename, the persona body's own `maxTurns: 40` citation is renamed to `steps: 40` so the shipped roster's prose matches the key the harness actually reads. - t248: persona pins now scope to the Turn Budget section, assert the worst-case-no-output semantics and number sync, and add a negative pin (t221-style): no harness names and no steps: key inside the core persona section. The opencode dist translation pins (steps: 40 present, no maxTurns leak, no other agent capped) are unchanged. - CHANGELOG 2.5.14: the persona bullet now describes the section as deliberately harness-neutral, with the enforcement matrix living in the docs. References upstream issue awslabs#609.
…uard Review feedback on PR awslabs#613 (apackeer) found a real protocol gap: the missing-verdict guard's step 3 check ("no `## Review` section for THIS iteration") relied on the reviewer-authored iteration number to detect staleness, but the conductor never supplies an identifier to check against. On the Part 0 revision path - READY at iteration 1, human rejects at the gate, builder revises a produces[] artifact, re-review runs "as at first entry" and is AGAIN iteration 1 - a re-review that is itself cut off before writing leaves the pre-revision artifact's `Iteration: 1, verdict READY` section sitting under the same heading, and the guard would misread it as covering the revision: exactly the silently-missing-verdict outcome this PR set out to eliminate. Fix (option (a) from the review): §12a step 1 now renames any existing `## Review` section to `## Review (superseded)` before EVERY reviewer dispatch, first entry or re-dispatch alike. Step 3's check no longer infers staleness from the iteration number - it no longer needs to, since a missing/current `## Review` section now means "incomplete review" uniformly on both paths. Mirrored in the core protocol and all five harness SKILL.md reviewer bullets (dist regenerated). Also addresses the P2 finding that a missing-verdict finding wasted a full builder dispatch on a defect the builder cannot act on (the artifact was never touched): step 3's missing-verdict branch now skips the lead-agent re-run and returns straight to re-invoking the reviewer, still counting the iteration. Ordinary NOT-READY (with real artifact findings) keeps the existing builder loop. t248 (renamed from t243 - the number collided with unrelated merged work during rebase, and t248 is unused) gains a dedicated test reproducing the stale-READY-after-revision scenario and pinning the supersede-before-dispatch fix, plus updated assertions for the skip-builder-reinvoke wording; the prior test's phrasing pins ("for THIS iteration", "check the iteration number") are replaced throughout core + all five harness SKILL.md files + dist. Smaller fixes from the same review round: - harness/codex/emit.ts: the reviewer personas' Turn Budget prose cited "the maxTurns: 40 frontmatter above" as the number-sync anchor, but Codex TOML personas have no frontmatter at all (Codex agent discovery reads only the TOML) and ship no cap key. emitAgentToml now rewrites that citation for the TOML surface, mirroring the opencode emitter's own prose rename for its `steps:` key. - docs/harness-engineering/03-adding-an-agent.md: the frontmatter contract list now names the optional `maxTurns` key. - docs/reference/04-stage-protocol.md, docs/guide/06-agents.md, docs/guide/glossary.md, docs/reference/agents/README.md: reworded to match the supersede-before-dispatch behavior and the skip-builder short-circuit. Investigated and confirmed safe (no code change): kiro-ide's agent .md frontmatter parser only reads the `tools:` grant line (harness/kiro-ide/manifest.ts frontmatterAdditions) and performs no generic unknown-key validation; `maxTurns` ships there as an inert, already-precedented unknown key (disallowedTools and display_name are already unrecognized-but-tolerated on that exact surface, field-proven on Kiro IDE 1.0.165+). References upstream issue awslabs#609, review on PR awslabs#613.
591f66a to
3e88ec0
Compare
|
Thanks for the thorough review — addressed the P1 and rebased. Point by point: 1. P1 — stale READY on the Part 0 revision path. Confirmed the gap: the old guard's "check the iteration number on any existing 2. P1 — rebase + slot renumbering. Rebased onto current 3. P2 — wasted builder dispatch on a missing-verdict finding. Fixed — small prose change. Step 3's missing-verdict branch now skips the lead-agent re-run and returns straight to re-invoking the reviewer (the artifact was never touched, so there's nothing for the builder to act on), still counting the iteration and still respecting 4. P2 — Codex TOML citing nonexistent frontmatter. Confirmed — Codex TOML personas have no frontmatter and no cap key (Codex agent discovery reads only the TOML). 5. P3 — agent-authoring guide missing 6. P3 — kiro-ide frontmatter parser verify-item. Investigated, no code change needed. Verification after rebase + fixes:
PR title updated to reflect 2.5.14. Ready for another pass. |
leandrodamascena
left a comment
There was a problem hiding this comment.
Thanks for this, Jeff — the targeted tests, package parity, typecheck, coverage registry, and the opencode steps: 40 resolution are all green, and I confirmed the supersede-before-dispatch rule genuinely closes the stale-verdict read. I saw a few semantic paths that the text-based tests don't exercise, though, and two of them can strand a workflow. Grouping by what I'd block on vs. what's worth fixing alongside:
Blockers
[P1] A cut-off review never records the terminal receipt. stage-protocol.md:993 dropped the old instruction to log aidlc-log.ts review --verdict <READY|NOT-READY>. On the missing-review path there's no verdict to "read," so nothing clearly emits a synthetic NOT-READY receipt — and after exhaustion verifyReviewerPrecondition() rejects the human's Approve because no fresh REVIEW_COMPLETED exists. Re-invoking just cuts off again, so the stage strands. Please record a synthetic NOT-READY receipt on every incomplete attempt. (The "reaches the human gate" guarantee itself is pre-existing — byte-identical at the merge base — so this is specifically about the new cutoff routing lacking the receipt that route needs.)
[P1] A partial ## Review leaves an undefined branch. The guard only checks whether the heading exists. A reviewer can write ## Review and then be cut off before writing a valid READY/NOT-READY verdict — that skips the missing-section branch (the heading is there) but satisfies neither verdict arm at :994/:995. Please require exactly one current section with exactly one canonical verdict; missing, malformed, multiple, or conflicting verdicts should all take the incomplete-review path.
Worth fixing alongside
[P2] The 40-turn default rests on the rejected effort step-down. The field baseline under inherited xhigh was 56–113 turns; the evidence that reviews finish under 40 came only after the medium effort change from #612, which was rejected. With the #546 contract preserved, a 40-turn default may terminate otherwise-normal reviews most of the time. Could you recalibrate and validate the cap under inherited session effort, or make it opt-in/configurable?
[P2] Reviewer knowledge conflicts with supersede-before-dispatch. Both reviewing.md files still say to "Update the ## Review section (replace, don't append a second one)" (arch :104, product-lead :82). But the conductor now renames the prior section to ## Review (superseded) and expects the reviewer to preserve it and append one new current ## Review. Please update those instructions and add coverage for the interaction.
Nit
[P3] The missing-review retry is bounded, but only implicitly. The cutoff branch does count the attempt and the "(iterations-exhausted still applies unchanged)" clause does terminate it — so it's not an infinite loop. But the exhaustion condition lives in the normal-verdict branch and the cutoff branch reaches it only via that parenthetical. Worth branching explicitly: retry only while attempts remain, otherwise the unresolved-findings path.
Also minor, not code: the PR body cites t243 (the file is t248; t243 is doctor-bundle) and "2.5.2" (shipped is 2.5.14) — and 2.5.14 collides with #661, so second-to-merge re-bumps.
|
Round 2, reviewed at head 3e88ec0. All six round-1 findings are fixed and verified - the supersede-before-dispatch rule closes the revision-path hole cleanly, and t248's negative pins are exactly what I asked for. One new P1 that the rebase introduced, one P2 interaction with the claim-sources sensor, and the usual slot note. Findings ranked below. Round-1 findings: all verified fixed
Independently re-verified at your head: New findings1. P1 - the rebase dropped the terminal-receipt sentence from section 12a step 3, and the new cutoff branch never records a receipt. On current v2, step 3 reads "...then read the This is load-bearing, not prose: Concrete failure: both iterations get capped; the new branch says iterations-exhausted applies unchanged; the protocol proceeds to the gate with unresolved findings noted; the human approves; the engine refuses the approve because zero Asks: (a) restore the terminal-receipt sentence in step 3; (b) state explicitly on the missing-verdict branch that the conductor records the receipt with 2. P2 - The claim-sources sensor skips exactly the H2 named Two ways out: teach the sensor to skip any H2 starting with 3. P3 - version slot. 2.5.14 is contested (#661 also claims it, and #617/#616/#667 hold 2.5.15/2.5.16). Standard policy applies: whichever merges second rebases and re-bumps the version, CHANGELOG heading, and badge. Flagging so it is not a surprise at merge time; the test slot t248 has a parallel collision risk for the same reason. Everything else I re-checked is clean: the opencode roster carries |
|
Re the request for a much higher turn limit: I posted field timing data on #612 (#612 (comment)) that sizes this cap, and it shows the two PRs really only work as a pair — I'd suggest deciding them together. The short version: at reviewer effort medium there's a clean gap between normal and runaway reviews (p90 40 turns, max 60 across 132 observed reviews), so the 40–60 range is a real backstop. At inherited session effort there is no such gap — legitimate reviews reached 116 turns (p90 112), so 40 truncates a third of them and a "safe" ~120 cap protects against essentially nothing. If the |
|
Thanks, Jeff. I want to make sure we evaluate these two PRs against the combined goal you described. Medium reviewer effort is the primary latency control: in your field data it reduced the architecture reviewer's median from 42.0 minutes to 4.1 minutes and p90 from 102.7 minutes to 14.6 minutes. The turn limit is then a deterministic backstop for outliers, while the incomplete-review guard makes a cap, crash, or harness cutoff fail clearly instead of leaving an undefined verdict path. We closed #612 because changing the existing @leandrodamascena, could we decide the following here?
The current guard still needs to validate exactly one canonical verdict and record a terminal NOT-READY receipt for an incomplete attempt, but that is an implementation detail after the policy decision. Jeff, please do not rebase or revise this branch while we resolve that direction. |
|
@jstrunk Reviewer tier has now been implemented. i will rebase and merge this. @leandrodamascena your request changes remains, ill request rereview when ready to merge |
Closes #609.
What
Bounds the §12a reviewer's work loop and closes the protocol's verdict-less-review branch. Three coordinated parts:
aidlc-architecture-reviewer-agent,aidlc-product-lead-agent), authored once asmaxTurns: 40in core and projected onto every harness that has a native cap (see the enforcement matrix below): Claude Code ships the frontmatter verbatim; opencode's emitter translates it to the nativesteps: 40(same seam as the existing tier → model/variant projection). Deliberately absent from Kiro agent JSON (fail-closed schema) and Codex TOML — neither harness has a cap key to project onto.## Turn Budgetpersona section in both reviewers, voiced per persona and deliberately harness-neutral: it assumes the hard cap is always enforced and plans for the worst-case cutoff (stopped mid-task, no warning, no final-message turn), reserves the FINAL ~5 turns for writing the## Reviewsection, and states that a verdict backed by fewer verified findings always beats no verdict. No harness names appear in the personas — the model behaves identically everywhere, and the per-harness enforcement truth lives in the docs matrix below (t243 pins the absence of harness names as a regression guard).## Reviewsection for THIS iteration, treat the result as NOT-READY with the finding "review did not complete within its turn budget" and count the iteration againstreviewer_max_iterations. The existing lead-alone loop and iterations-exhausted paths apply unchanged.Part 3 stands alone even without the cap — the undefined branch was reachable today by any reviewer death (crash, context exhaustion).
Per-harness enforcement matrix
Researched against each harness's current docs and source (2026-07-20):
maxTurnssteps(v1.0.134+; legacymaxStepsdeprecated).opencode/agents/*.mdfrontmatter — projected from the authoredmaxTurnsby the emitter## Reviewsection; the missing-verdict guard stays load-bearingfeatures.rollout_budget.limit_tokens, is token-based and session-tree-wide — unusable per-reviewer. Prose-only.additionalProperties: falsewith no cap field (verified against the pinned schema, kiro.dev docs, and the 2.12.x/2.13.0 binaries). Kiro v3's new iteration caps (/goal --max N, agent-crewloop_config.max_iterations) are orchestration surfaces, not agent config. Prose-only.The matrix is maintainer/user-facing documentation only — the personas deliberately do not mention it. On the opencode roster the emitter also renames the persona's backticked
maxTurns: 40citation tosteps: 40, so each shipped tree's prose names its own frontmatter key (the "two numbers in sync" contract holds per tree).Why
The read-scope bound (#534 → PR #538, #539) limits WHERE the reviewer reads; nothing bounded HOW MUCH it does — field reviews ran 56–113 turns each under an xhigh session. Probe-verified cutoff semantics made shipping a naked cap dangerous: the reviewer writes its verdict as one Edit at the END of investigation, so an uninstructed cap silently produces verdict-less reviews the conductor then misreads. With the guard, a cut-off review degrades gracefully: it consumes an iteration, the builder gets a concrete finding, and the gate is never presented on a silently-missing verdict. Full data and semantics in #609.
Changes
core/agents/aidlc-architecture-reviewer-agent.md,core/agents/aidlc-product-lead-agent.md— frontmatter +## Turn Budget(per-harness enforcement sentences).harness/opencode/emit.ts—emitSubagentMdrenames a coremaxTurns:frontmatter line to the nativesteps:in the.opencode/agents/roster (the only directory opencode reads for agent config; the.aidlc/agents/engine-tree mirror keeps core shape, as pinned by t240).core/aidlc-common/protocols/stage-protocol.md§12a step 3 — purely additive branch after the dispatch-record deletion (t221's pinned ordering preserved).harness/{claude,kiro,kiro-ide,codex,opencode}/skills/aidlc/SKILL.md— one-sentence rule in the reviewer bullet (kiro-ide stays dispatch-record-free, as pinned).tests/unit/t243-reviewer-turn-budget.test.ts— 8 tests / 119 assertions pinning the frontmatter (core + dist), the number sync between frontmatter and prose, the worst-case cutoff semantics, the missing-verdict sentence across §12a + all five SKILLs, the opencode translation (exactly onesteps: 40on each reviewer in the native roster, nomaxTurnsleak,stepsabsent from the other 12 agents), and a negative pin that the persona Turn Budget sections contain no harness names.docs/reference/04-stage-protocol.md(full enforcement matrix),docs/guide/06-agents.md,docs/guide/glossary.md,docs/reference/agents/README.md.Testing
bun scripts/package.ts --checkand the coverage-registry check both clean.Notes for review