Skip to content

feat: reviewer turn backstop (maxTurns) + §12a missing-verdict guard (2.5.14) - #613

Open
jstrunk wants to merge 4 commits into
awslabs:v2from
jstrunk:feat/609-reviewer-turn-budget
Open

feat: reviewer turn backstop (maxTurns) + §12a missing-verdict guard (2.5.14)#613
jstrunk wants to merge 4 commits into
awslabs:v2from
jstrunk:feat/609-reviewer-turn-budget

Conversation

@jstrunk

@jstrunk jstrunk commented Jul 20, 2026

Copy link
Copy Markdown

Closes #609.

What

Bounds the §12a reviewer's work loop and closes the protocol's verdict-less-review branch. Three coordinated parts:

  1. A turn cap on both balanced reviewer agents (aidlc-architecture-reviewer-agent, aidlc-product-lead-agent), authored once as maxTurns: 40 in core and projected onto every harness that has a native cap (see the enforcement matrix below): Claude Code ships the frontmatter verbatim; opencode's emitter translates it to the native steps: 40 (same seam as the existing tier → model/variant projection). Deliberately absent from Kiro agent JSON (fail-closed schema) and Codex TOML — neither harness has a cap key to project onto.
  2. ## Turn Budget persona section in both reviewers, voiced per persona and deliberately harness-neutral: it assumes the hard cap is always enforced and plans for the worst-case cutoff (stopped mid-task, no warning, no final-message turn), reserves the FINAL ~5 turns for writing the ## Review section, and states that a verdict backed by fewer verified findings always beats no verdict. No harness names appear in the personas — the model behaves identically everywhere, and the per-harness enforcement truth lives in the docs matrix below (t243 pins the absence of harness names as a regression guard).
  3. Missing-verdict guard in stage-protocol §12a step 3 + one sentence in all five harness SKILL.md reviewer bullets: if the primary artifact has no ## Review section for THIS iteration, treat the result as NOT-READY with the finding "review did not complete within its turn budget" and count the iteration against reviewer_max_iterations. The existing lead-alone loop and iterations-exhausted paths apply unchanged.

Part 3 stands alone even without the cap — the undefined branch was reachable today by any reviewer death (crash, context exhaustion).

Per-harness enforcement matrix

Researched against each harness's current docs and source (2026-07-20):

Harness Native cap Surface Cutoff semantics
Claude Code maxTurns agent .md frontmatter (documented subagent field) Hard stop mid-task; no final message — caller receives no output
opencode steps (v1.0.134+; legacy maxSteps deprecated) .opencode/agents/*.md frontmatter — projected from the authored maxTurns by the emitter Final turn forced text-only: the agent can return a summary but cannot make tool calls — so it still cannot write the ## Review section; the missing-verdict guard stays load-bearing
Codex CLI ❌ none No turn/iteration/tool-call cap key exists in config.toml, role TOMLs, CLI flags, or the SDK (verified at rust-v0.144.6). Nearest control, features.rollout_budget.limit_tokens, is token-based and session-tree-wide — unusable per-reviewer. Prose-only.
Kiro CLI/IDE ❌ none agent-v1 schema is additionalProperties: false with no cap field (verified against the pinned schema, kiro.dev docs, and the 2.12.x/2.13.0 binaries). Kiro v3's new iteration caps (/goal --max N, agent-crew loop_config.max_iterations) are orchestration surfaces, not agent config. Prose-only.

The matrix is maintainer/user-facing documentation only — the personas deliberately do not mention it. On the opencode roster the emitter also renames the persona's backticked maxTurns: 40 citation to steps: 40, so each shipped tree's prose names its own frontmatter key (the "two numbers in sync" contract holds per tree).

Why

The read-scope bound (#534 → PR #538, #539) limits WHERE the reviewer reads; nothing bounded HOW MUCH it does — field reviews ran 56–113 turns each under an xhigh session. Probe-verified cutoff semantics made shipping a naked cap dangerous: the reviewer writes its verdict as one Edit at the END of investigation, so an uninstructed cap silently produces verdict-less reviews the conductor then misreads. With the guard, a cut-off review degrades gracefully: it consumes an iteration, the builder gets a concrete finding, and the gate is never presented on a silently-missing verdict. Full data and semantics in #609.

Changes

  • core/agents/aidlc-architecture-reviewer-agent.md, core/agents/aidlc-product-lead-agent.md — frontmatter + ## Turn Budget (per-harness enforcement sentences).
  • harness/opencode/emit.tsemitSubagentMd renames a core maxTurns: frontmatter line to the native steps: in the .opencode/agents/ roster (the only directory opencode reads for agent config; the .aidlc/agents/ engine-tree mirror keeps core shape, as pinned by t240).
  • core/aidlc-common/protocols/stage-protocol.md §12a step 3 — purely additive branch after the dispatch-record deletion (t221's pinned ordering preserved).
  • harness/{claude,kiro,kiro-ide,codex,opencode}/skills/aidlc/SKILL.md — one-sentence rule in the reviewer bullet (kiro-ide stays dispatch-record-free, as pinned).
  • New tests/unit/t243-reviewer-turn-budget.test.ts — 8 tests / 119 assertions pinning the frontmatter (core + dist), the number sync between frontmatter and prose, the worst-case cutoff semantics, the missing-verdict sentence across §12a + all five SKILLs, the opencode translation (exactly one steps: 40 on each reviewer in the native roster, no maxTurns leak, steps absent from the other 12 agents), and a negative pin that the persona Turn Budget sections contain no harness names.
  • Docs: docs/reference/04-stage-protocol.md (full enforcement matrix), docs/guide/06-agents.md, docs/guide/glossary.md, docs/reference/agents/README.md.
  • Version 2.5.2 + CHANGELOG entry + README badge; all dists regenerated.

Testing

  • Full smoke + unit tiers (after the opencode projection commit): 174 files, 4264 assertions, 0 failures.
  • Targeted t221 / t217 / t234 / t200 / t68 / t243: 90 tests, 347 assertions, 0 failures; opencode adapter suites t240 / t241 + t243: 29 pass, 0 fail.
  • Adjacent doc-parity and conformance suites (t239, t216, t148, t123): 1013 pass, 0 fail.
  • bun scripts/package.ts --check and the coverage-registry check both clean.

Notes for review

jstrunk added a commit to jstrunk/aidlc-workflows that referenced this pull request Jul 20, 2026
…ays enforced

Review feedback on PR awslabs#613: persona prose in core/agents/*.md should
not carry harness-specific instructions, and telling the model "on
Codex and Kiro no native cap exists; this is prose you hold yourself
to" undermines the budget on exactly the harnesses without
enforcement.

- Both reviewers' '## Turn Budget' enforcement bullet is rewritten
  unconditionally: a HARD cap of 40 turns; when hit you are stopped,
  in the worst case with NO final message and NO chance to write
  anything - so plan for the worst case, protect the reserve, and
  write the review before the cap, never on the last turn. All
  harness names and per-harness cutoff semantics removed; the
  frontmatter/prose number-sync note stays (harness-free wording);
  per-persona voicing kept.
- The per-harness enforcement matrix remains where it belongs:
  docs/reference/04-stage-protocol.md and docs/reference/agents/
  README.md (maintainer/user-facing), unchanged.
- opencode emit: alongside the maxTurns->steps frontmatter rename,
  the persona body's own `maxTurns: 40` citation is renamed to
  `steps: 40` so the shipped roster's prose matches the key the
  harness actually reads.
- t243: persona pins now scope to the Turn Budget section, assert the
  worst-case-no-output semantics and number sync, and add a negative
  pin (t221-style): no harness names and no steps: key inside the
  core persona section. The opencode dist translation pins
  (steps: 40 present, no maxTurns leak, no other agent capped) are
  unchanged.
- CHANGELOG 2.5.2: the persona bullet now describes the section as
  deliberately harness-neutral, with the enforcement matrix living in
  the docs.

References upstream issue awslabs#609.
@jstrunk
jstrunk marked this pull request as ready for review July 20, 2026 23:48
@apackeer

Copy link
Copy Markdown
Contributor

Reviewed at head 591f66a. Verdict: the design is sound and the external-harness claims all check out against primary sources. One real protocol gap on the revision path (finding 1 below) and the usual rebase/slot logistics (finding 2 below) before this can land; the rest are smaller. Findings ranked.

Verified first (primary sources, not the PR text)

  • maxTurns is a documented Claude Code subagent frontmatter field ("Maximum number of agentic turns before the subagent stops"). The official docs do NOT document the cutoff semantics, so the "no final message, caller receives no output" claim rests on the [Feature]: reviewer turn backstop (maxTurns) + §12a missing-verdict guard — bound the reviewer's tool-call count and close the verdict-less-review branch #609 probe, which is disclosed there. Acceptable evidence.
  • The opencode claims verified against the shipped 1.17.18 binary: the agent config schema carries steps ("Maximum number of agentic iterations before forcing text-only response") and maxSteps annotated "@deprecated Use 'steps' field instead". So the emitter rename and the "final text-only turn cannot write the review" reasoning are both accurate, and the missing-verdict guard genuinely stays load-bearing there.
  • The emitter change is well anchored: the frontmatter rename matches only a ^maxTurns: <n>$ line and the prose rename only backticked maxTurns: <n> citations, so it cannot munge arbitrary body text. .opencode/agents/ is correctly the only translated surface; the .aidlc/agents/ mirror keeps core shape, so each shipped tree's prose cites its own frontmatter key.
  • The missing-verdict sentence is present in all five authored SKILL.md files plus dist copies; kiro-ide stays dispatch-record-free (t221's pin holds); the section 12a step 3 edit preserves the delete-record-then-read ordering.
  • The iteration-number check binds to a real contract: the prescribed review format carries **Iteration:** [1, 2, etc.] plus "replace, don't append" in both reviewers' reviewing.md knowledge.
  • At the PR head as authored: package drift check, typecheck, coverage registry, and the smoke (1117 assertions) + unit (3147 assertions) tiers all green.

Findings

1. P1 - the guard still accepts a stale READY on the Part 0 revision path.

Sequence: a stage passes review READY at iteration 1; the human rejects at the gate; the builder revises a produces[] artifact; section 12a re-runs "as at first entry" with a fresh iteration budget, so the re-review is again iteration 1. If that re-review is capped before writing (the exact failure this PR targets), the artifact still carries the pre-revision section reading Iteration: 1, verdict READY. The new guard's "check the iteration number on any existing ## Review section" matches, so the conductor reads a stale READY that predates the revision as current and presents the gate on it - precisely the silently-missing-verdict outcome the PR set out to eliminate, still reachable on the revision path. Note the iteration number in the template is reviewer-inferred; the conductor supplies no identifier to check against.

Two ways to close it: (a) the conductor removes or renames the stale section (e.g. ## Review (superseded)) before every reviewer re-dispatch - then a missing section means an incomplete review on every path, and the iteration check becomes belt-and-braces rather than the sole discriminator (the "does not modify the artifact" constraint binds the reviewer, not the conductor, so no conflict); or (b) the conductor passes a unique attempt token into the dispatch and the reviewer echoes it in the section. (a) is the smaller change: one conductor-side sentence in section 12a step 1 plus the SKILL bullets. Whichever lands, please add a test that pins this failure path - t243 currently pins the wording of the guard, not its semantics.

2. P1 - rebase + slot renumbering.

The branch is CONFLICTING against v2, and both of its slots are now taken: v2 is at 2.5.2 (#576 merged, heading ## [2.5.2] - 2026-07-21) and tests/unit/t243-doctor-bundle.test.ts occupies t243. The branch also predates 2.5.1 (the CHANGELOG diff inserts 2.5.2 directly above 2.5.0). Needed: rebase onto v2; re-bump version + CHANGELOG heading + README badge to the next free slot at merge time (coordinate with the other open claims: #617 has 2.5.3, #616 has 2.5.4, #569 has 2.5.5); rename the test to t244 (free on v2) including its covers: header; regenerate dists and the coverage registry; re-run the reported tiers. t68 will red on the duplicate ## [2.5.2] heading immediately post-rebase, so this is mechanical but mandatory.

3. P2 - the missing-verdict branch spends a full builder dispatch on a finding the builder cannot act on.

"review did not complete within its turn budget" feeds the existing NOT-READY loop, whose first step re-invokes the lead alone to address the findings - but this finding is not an artifact defect; there is nothing for the builder to fix. Consider short-circuiting this specific finding straight to the reviewer re-invoke (still counting the iteration). Not blocking - the current shape is simple and bounded - but it wastes a whole subagent run per cutoff.

4. P2 - the Codex TOML persona points at frontmatter that does not exist on that surface.

dist/codex/.codex/agents/*.toml carry "the maxTurns: 40 frontmatter above - keep the two numbers in sync" inside developer_instructions, which has no frontmatter and no cap key (Codex agent discovery reads only the TOML; the .md beside it is conductor-side prose). The harness-neutral stance is fine as disclosed, but this sentence is a concrete false pointer, and the keep-in-sync maintenance instruction is addressed to a model that cannot see the file it names. The opencode emitter already solved exactly this pattern with its prose rename; the Codex emitter could rewrite or drop the parenthetical when transposing to TOML.

5. P3 - the agent-authoring guide omits the new key.

docs/harness-engineering/03-adding-an-agent.md enumerates the authored frontmatter contract (name, display_name, examples, description, disallowedTools, tier, optional tools:) and now misses the optional maxTurns. One line there keeps the documentation policy satisfied.

6. P3 - verify-item: the kiro-ide .md frontmatter parser.

The kiro-ide dist personas now carry an unknown frontmatter key on the surface the IDE actually parses for subagent tool grants (the frontmatterAdditions seam in the kiro-ide manifest). Almost certainly ignored as an unknown key, but the PR's schema research covered the agent-v1 JSON, not the IDE's .md frontmatter parser - one sentence of confirmation from your kiro-ide probing would close it.

On the test (t243, to become t244)

Good coverage shape: both sides of the number-sync contract, the worst-case-cutoff wording, the harness-name negative pin, the opencode translation (exactly one steps: 40 per reviewer, no maxTurns leak, no cap on the other 12 agents), the section 12a branch ordering after t221's pin, and all five SKILL mirrors via HARNESS_MATRIX. The roster-count pin (14) will need a touch when an agent lands, consistent with existing pins. No asks beyond the renumbering and the stale-READY semantics test asked in finding 1.

jstrunk added 4 commits July 28, 2026 16:00
Field reviews ran 56-113 turns each; nothing bounded how much work a
§12a reviewer did inside its read scope, and a reviewer stopped before
writing its verdict left the conductor on an undefined branch. Three
coordinated changes (upstream issue awslabs#609):

- maxTurns: 40 in both reviewer agents' frontmatter (binding on Claude
  Code; ships inertly in the kiro/opencode/codex .md copies - the
  packager's projectTierFrontmatter rewrites only the tier: line, so
  the key passes through verbatim; not added to Kiro agent JSON or the
  Codex TOML headers).
- '## Turn Budget' persona section in both reviewers, voiced
  per-persona: cutoff semantics (stopped mid-task, no warning, no
  final-message turn, caller receives no output), the ~15 read / ~5
  validation / ~10 verify / final ~5 reserved-for-writing split, and
  the standing order that a verdict backed by fewer verified findings
  always beats no verdict.
- §12a step 3 missing-verdict branch (additive after the
  dispatch-record deletion t221 pins): a primary artifact with no
  '## Review' section for THIS iteration reads as NOT-READY with the
  finding "review did not complete within its turn budget" and counts
  against reviewer_max_iterations; the existing loop and
  iterations-exhausted paths apply unchanged. Mirrored in all five
  harness SKILL.md reviewer bullets (kiro-ide stays
  dispatch-record-free).

New pin test tests/unit/t248-reviewer-turn-budget.test.ts guards the
frontmatter cap, the number sync with the persona prose, and the
missing-verdict sentence on every surface (core + dist). Docs updated
(04-stage-protocol, guide/06-agents, glossary, reference/agents
README). Version 2.5.14 + CHANGELOG + README badge; dist regenerated
via bun scripts/package.ts (--check green).

References upstream issue awslabs#609.
…ss enforcement notes

Research (2026-07-20, official docs + source): opencode has a native
per-agent step cap - `steps` (positive integer, agent .md frontmatter
or opencode.json; legacy `maxSteps` deprecated; shipped since
v1.0.134). Its cutoff differs from Claude Code's: the runner forces a
final TEXT-ONLY turn (tools disabled, "MAXIMUM STEPS REACHED"), so the
agent returns a summary but cannot make the tool call that writes the
'## Review' section - the §12a missing-verdict guard still matters
there. Codex CLI has no turn/iteration cap key (nearest is
features.rollout_budget.limit_tokens - token-based, session-tree-wide;
verified against the config reference and a rust-v0.144.6 source
grep). Kiro CLI/IDE agent-v1 is additionalProperties:false with no cap
field (v3/KAS adds caps only to /goal --max and agent_crew loop_config
- orchestration surfaces, not agent JSON).

- harness/opencode/emit.ts (emitSubagentMd): rename a core `maxTurns:`
  frontmatter line to opencode's native `steps:` in the .opencode/
  native roster (the only dir opencode reads) - same seam as the
  existing tier->model/variant projection. The authored core key stays
  maxTurns (single authored source).
- Turn Budget persona sections (both reviewers): per-harness accurate
  enforcement sentence - hard-capped on Claude Code (stopped mid-task,
  no final message) AND opencode (final turn text-only: summary yes,
  review write no); prose-only on Codex and Kiro (no native cap
  exists).
- Docs: 04-stage-protocol, guide/06-agents, glossary, and
  reference/agents README now state the per-harness enforcement matrix
  with one-line reasons for the Codex/Kiro absence.
- t248: new pin - opencode dist reviewers carry `steps: 40` with no
  `maxTurns:` leak, and no other opencode agent gains a steps cap;
  cutoff-semantics pin loosened to per-persona voicing.
- CHANGELOG 2.5.14 bullets updated to the researched matrix (version
  stays 2.5.14).

References upstream issue awslabs#609.
…ays enforced

Review feedback on PR awslabs#613: persona prose in core/agents/*.md should
not carry harness-specific instructions, and telling the model "on
Codex and Kiro no native cap exists; this is prose you hold yourself
to" undermines the budget on exactly the harnesses without
enforcement.

- Both reviewers' '## Turn Budget' enforcement bullet is rewritten
  unconditionally: a HARD cap of 40 turns; when hit you are stopped,
  in the worst case with NO final message and NO chance to write
  anything - so plan for the worst case, protect the reserve, and
  write the review before the cap, never on the last turn. All
  harness names and per-harness cutoff semantics removed; the
  frontmatter/prose number-sync note stays (harness-free wording);
  per-persona voicing kept.
- The per-harness enforcement matrix remains where it belongs:
  docs/reference/04-stage-protocol.md and docs/reference/agents/
  README.md (maintainer/user-facing), unchanged.
- opencode emit: alongside the maxTurns->steps frontmatter rename,
  the persona body's own `maxTurns: 40` citation is renamed to
  `steps: 40` so the shipped roster's prose matches the key the
  harness actually reads.
- t248: persona pins now scope to the Turn Budget section, assert the
  worst-case-no-output semantics and number sync, and add a negative
  pin (t221-style): no harness names and no steps: key inside the
  core persona section. The opencode dist translation pins
  (steps: 40 present, no maxTurns leak, no other agent capped) are
  unchanged.
- CHANGELOG 2.5.14: the persona bullet now describes the section as
  deliberately harness-neutral, with the enforcement matrix living in
  the docs.

References upstream issue awslabs#609.
…uard

Review feedback on PR awslabs#613 (apackeer) found a real protocol gap: the
missing-verdict guard's step 3 check ("no `## Review` section for THIS
iteration") relied on the reviewer-authored iteration number to detect
staleness, but the conductor never supplies an identifier to check
against. On the Part 0 revision path - READY at iteration 1, human
rejects at the gate, builder revises a produces[] artifact, re-review
runs "as at first entry" and is AGAIN iteration 1 - a re-review that
is itself cut off before writing leaves the pre-revision artifact's
`Iteration: 1, verdict READY` section sitting under the same heading,
and the guard would misread it as covering the revision: exactly the
silently-missing-verdict outcome this PR set out to eliminate.

Fix (option (a) from the review): §12a step 1 now renames any existing
`## Review` section to `## Review (superseded)` before EVERY reviewer
dispatch, first entry or re-dispatch alike. Step 3's check no longer
infers staleness from the iteration number - it no longer needs to,
since a missing/current `## Review` section now means "incomplete
review" uniformly on both paths. Mirrored in the core protocol and all
five harness SKILL.md reviewer bullets (dist regenerated).

Also addresses the P2 finding that a missing-verdict finding wasted a
full builder dispatch on a defect the builder cannot act on (the
artifact was never touched): step 3's missing-verdict branch now skips
the lead-agent re-run and returns straight to re-invoking the reviewer,
still counting the iteration. Ordinary NOT-READY (with real artifact
findings) keeps the existing builder loop.

t248 (renamed from t243 - the number collided with unrelated merged
work during rebase, and t248 is unused) gains a dedicated test
reproducing the stale-READY-after-revision scenario and pinning the
supersede-before-dispatch fix, plus updated assertions for the
skip-builder-reinvoke wording; the prior test's phrasing pins
("for THIS iteration", "check the iteration number") are replaced
throughout core + all five harness SKILL.md files + dist.

Smaller fixes from the same review round:
- harness/codex/emit.ts: the reviewer personas' Turn Budget prose cited
  "the maxTurns: 40 frontmatter above" as the number-sync anchor, but
  Codex TOML personas have no frontmatter at all (Codex agent discovery
  reads only the TOML) and ship no cap key. emitAgentToml now rewrites
  that citation for the TOML surface, mirroring the opencode emitter's
  own prose rename for its `steps:` key.
- docs/harness-engineering/03-adding-an-agent.md: the frontmatter
  contract list now names the optional `maxTurns` key.
- docs/reference/04-stage-protocol.md, docs/guide/06-agents.md,
  docs/guide/glossary.md, docs/reference/agents/README.md: reworded to
  match the supersede-before-dispatch behavior and the skip-builder
  short-circuit.

Investigated and confirmed safe (no code change): kiro-ide's agent .md
frontmatter parser only reads the `tools:` grant line
(harness/kiro-ide/manifest.ts frontmatterAdditions) and performs no
generic unknown-key validation; `maxTurns` ships there as an inert,
already-precedented unknown key (disallowedTools and display_name are
already unrecognized-but-tolerated on that exact surface, field-proven
on Kiro IDE 1.0.165+).

References upstream issue awslabs#609, review on PR awslabs#613.
@jstrunk
jstrunk force-pushed the feat/609-reviewer-turn-budget branch from 591f66a to 3e88ec0 Compare July 28, 2026 16:03
@jstrunk jstrunk changed the title feat: reviewer turn backstop (maxTurns) + §12a missing-verdict guard (2.5.2) feat: reviewer turn backstop (maxTurns) + §12a missing-verdict guard (2.5.14) Jul 28, 2026
@jstrunk

jstrunk commented Jul 28, 2026

Copy link
Copy Markdown
Author

Thanks for the thorough review — addressed the P1 and rebased. Point by point:

1. P1 — stale READY on the Part 0 revision path. Confirmed the gap: the old guard's "check the iteration number on any existing ## Review section" had no conductor-supplied identifier to check against, so a re-review cut off before writing could leave the pre-revision Iteration: 1, verdict READY section looking current. Implemented option (a): §12a step 1 now renames any existing ## Review section to ## Review (superseded) before every reviewer dispatch — first entry or re-dispatch alike — so step 3's check no longer infers staleness from the iteration number at all; "no current ## Review section" now means "incomplete review" uniformly on both paths. Updated in core/aidlc-common/protocols/stage-protocol.md §12a (steps 1 and 3, plus the Part 0 revision-path paragraph) and mirrored in all five harness SKILL.md reviewer bullets. Added a dedicated test in the renamed t248-reviewer-turn-budget.test.ts that reproduces the stale-READY-after-revision scenario and pins the supersede-before-dispatch fix (plus a negative pin that the old iteration-number-inference phrasing is gone).

2. P1 — rebase + slot renumbering. Rebased onto current v2 (9f91454). Used your pre-assigned numbers: version 2.5.22.5.14, test file t243-reviewer-turn-budget.test.tst248-reviewer-turn-budget.test.ts. CHANGELOG heading is ## [2.5.14] - 2026-07-28, README badge and all six aidlc-version.ts copies updated.

3. P2 — wasted builder dispatch on a missing-verdict finding. Fixed — small prose change. Step 3's missing-verdict branch now skips the lead-agent re-run and returns straight to re-invoking the reviewer (the artifact was never touched, so there's nothing for the builder to act on), still counting the iteration and still respecting reviewer_max_iterations. Ordinary NOT-READY with real artifact findings keeps the existing builder loop unchanged.

4. P2 — Codex TOML citing nonexistent frontmatter. Confirmed — Codex TOML personas have no frontmatter and no cap key (Codex agent discovery reads only the TOML). harness/codex/emit.ts's emitAgentToml now rewrites the "the maxTurns: 40 frontmatter above — keep the two numbers in sync" citation to name the actual state on that surface (no frontmatter, no native cap key, prose-only number), following the same precedent as the opencode emitter's own maxTurns:steps: prose rename.

5. P3 — agent-authoring guide missing maxTurns. Added a line to docs/harness-engineering/03-adding-an-agent.md's frontmatter-contract list documenting the optional maxTurns: <n> key and its per-harness enforcement shape.

6. P3 — kiro-ide frontmatter parser verify-item. Investigated, no code change needed. harness/kiro-ide/manifest.ts's frontmatterAdditions seam only appends the tools: grant line to the persona .md frontmatter; there's no generic unknown-key validation anywhere in the packager for that surface. maxTurns ships there exactly like the pre-existing disallowedTools and display_name keys, which the IDE already tolerates as unrecognized-but-inert (field-proven on Kiro IDE 1.0.165+, per the 2.5.10 hook-registration work). Confirmed safe.

Verification after rebase + fixes:

  • bun scripts/package.ts --check — clean, all five harness trees in sync.
  • bun tests/gen-coverage-registry.ts --check — OK, ratchet held.
  • bun run check (package drift + typecheck + biome lint) — clean.
  • bash tests/run-tests.sh --smoke --unit — 179 test files, 4405+ assertions, 0 failures.
  • Targeted: t248-reviewer-turn-budget.test.ts (9 tests), t221-reviewer-scope-hook.test.ts, t68-version-changelog-sync.test.ts — all green.

PR title updated to reflect 2.5.14. Ready for another pass.

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, Jeff — the targeted tests, package parity, typecheck, coverage registry, and the opencode steps: 40 resolution are all green, and I confirmed the supersede-before-dispatch rule genuinely closes the stale-verdict read. I saw a few semantic paths that the text-based tests don't exercise, though, and two of them can strand a workflow. Grouping by what I'd block on vs. what's worth fixing alongside:

Blockers

[P1] A cut-off review never records the terminal receipt. stage-protocol.md:993 dropped the old instruction to log aidlc-log.ts review --verdict <READY|NOT-READY>. On the missing-review path there's no verdict to "read," so nothing clearly emits a synthetic NOT-READY receipt — and after exhaustion verifyReviewerPrecondition() rejects the human's Approve because no fresh REVIEW_COMPLETED exists. Re-invoking just cuts off again, so the stage strands. Please record a synthetic NOT-READY receipt on every incomplete attempt. (The "reaches the human gate" guarantee itself is pre-existing — byte-identical at the merge base — so this is specifically about the new cutoff routing lacking the receipt that route needs.)

[P1] A partial ## Review leaves an undefined branch. The guard only checks whether the heading exists. A reviewer can write ## Review and then be cut off before writing a valid READY/NOT-READY verdict — that skips the missing-section branch (the heading is there) but satisfies neither verdict arm at :994/:995. Please require exactly one current section with exactly one canonical verdict; missing, malformed, multiple, or conflicting verdicts should all take the incomplete-review path.

Worth fixing alongside

[P2] The 40-turn default rests on the rejected effort step-down. The field baseline under inherited xhigh was 56–113 turns; the evidence that reviews finish under 40 came only after the medium effort change from #612, which was rejected. With the #546 contract preserved, a 40-turn default may terminate otherwise-normal reviews most of the time. Could you recalibrate and validate the cap under inherited session effort, or make it opt-in/configurable?

[P2] Reviewer knowledge conflicts with supersede-before-dispatch. Both reviewing.md files still say to "Update the ## Review section (replace, don't append a second one)" (arch :104, product-lead :82). But the conductor now renames the prior section to ## Review (superseded) and expects the reviewer to preserve it and append one new current ## Review. Please update those instructions and add coverage for the interaction.

Nit

[P3] The missing-review retry is bounded, but only implicitly. The cutoff branch does count the attempt and the "(iterations-exhausted still applies unchanged)" clause does terminate it — so it's not an infinite loop. But the exhaustion condition lives in the normal-verdict branch and the cutoff branch reaches it only via that parenthetical. Worth branching explicitly: retry only while attempts remain, otherwise the unresolved-findings path.

Also minor, not code: the PR body cites t243 (the file is t248; t243 is doctor-bundle) and "2.5.2" (shipped is 2.5.14) — and 2.5.14 collides with #661, so second-to-merge re-bumps.

@apackeer

Copy link
Copy Markdown
Contributor

Round 2, reviewed at head 3e88ec0. All six round-1 findings are fixed and verified - the supersede-before-dispatch rule closes the revision-path hole cleanly, and t248's negative pins are exactly what I asked for. One new P1 that the rebase introduced, one P2 interaction with the claim-sources sensor, and the usual slot note. Findings ranked below.

Round-1 findings: all verified fixed

  1. Stale READY on the revision path - option (a) implemented: step 1 renames any existing ## Review section to ## Review (superseded) before every dispatch, mirrored in all five SKILL.md reviewer bullets; t248 pins the rule plus a negative pin that the iteration-number-inference phrasing is gone. Verified in stage-protocol.md and each harness SKILL.
  2. Rebase + slots - on 9f91454, 2.5.14 + t248, CHANGELOG heading / README badge / all six version copies agree (t68 green).
  3. Missing-verdict branch now skips the builder re-invoke and returns straight to the reviewer - confirmed in step 3 and all five SKILLs.
  4. Codex TOML citation - emitAgentToml rewrites the frontmatter pointer to the actual state on that surface; verified in both shipped TOMLs, and the .md twin beside them keeps the frontmatter-accurate wording.
  5. maxTurns documented in docs/harness-engineering/03-adding-an-agent.md.
  6. kiro-ide frontmatter - agreed: the key ships beside disallowedTools and display_name, which that surface already tolerates as unknown-but-inert.

Independently re-verified at your head: bun scripts/package.ts --check, typecheck + lint, the coverage-registry check, and the smoke + unit tiers (179 files, 4405 assertions, 0 failures), plus t248 (9 tests) targeted.

New findings

1. P1 - the rebase dropped the terminal-receipt sentence from section 12a step 3, and the new cutoff branch never records a receipt.

On current v2, step 3 reads "...then read the ## Review section from the primary artifact. Record the terminal receipt with the same aidlc-log.ts review command plus --verdict <READY|NOT-READY> (and the same --unit / --single fields), then branch on the verdict:". That sentence landed in 2.5.5 (#569), after this branch forked, and the step-3 rewrite here kept the pre-2.5.5 text - so the shipped protocol no longer instructs the conductor to write the terminal receipt at all.

This is load-bearing, not prose: verifyReviewerPrecondition in core/tools/aidlc-state.ts refuses every completion path (approve, advance, finalize, complete-workflow) for a reviewer-bearing stage until a fresh REVIEW_COMPLETED row exists. The SKILL bullets still carry their own copy of the receipt instruction, but it is phrased "after reading the verdict, run the same command with --verdict ..." - and on the missing-verdict branch there is no verdict to read, so no surface tells the conductor to record anything for a cut-off attempt.

Concrete failure: both iterations get capped; the new branch says iterations-exhausted applies unchanged; the protocol proceeds to the gate with unresolved findings noted; the human approves; the engine refuses the approve because zero REVIEW_COMPLETED rows exist for the stage. That is exactly the outcome the precondition's own comment wants to permit ("soft on the verdict: a NOT-READY-after-cap still lets the human approve") - but only when the receipt is recorded.

Asks: (a) restore the terminal-receipt sentence in step 3; (b) state explicitly on the missing-verdict branch that the conductor records the receipt with --verdict NOT-READY (the "after reading the verdict" phrasing does not reach this branch); (c) mirror in the five SKILL bullets if the wording changes; (d) add a pin - no test guards that sentence today, which is why the rebase could drop it silently. t248 is the natural home.

2. P2 - ## Review (superseded) sections lose the claim-sources exclusion and accumulate without bound.

The claim-sources sensor skips exactly the H2 named Review (REVIEW_HEADING = "Review", exact match in core/tools/aidlc-sensor-claim-sources.ts). After a supersede rename the heading no longer matches, so every untagged line of the old review body becomes a claim-sources finding. This fires on the very first NOT-READY loop of Intent Capture: iteration 2's dispatch renames iteration 1's section, and the sensor re-checks the deliverables on the next write. Advisory severity, so nothing blocks - but Intent Capture declares both this reviewer and this sensor, so the noise is guaranteed, not hypothetical. The renamed sections also accumulate across iterations and Part 0 revisions as duplicate identical headings.

Two ways out: teach the sensor to skip any H2 starting with Review (one line plus a manifest sentence), or have step 1 delete the stale section rather than rename it - the reviewers' own iteration contract is already "replace, don't append", so preserving superseded reviews in the artifact is new behavior rather than preserved history. Either works; deleting is smaller and keeps artifacts clean.

3. P3 - version slot. 2.5.14 is contested (#661 also claims it, and #617/#616/#667 hold 2.5.15/2.5.16). Standard policy applies: whichever merges second rebases and re-bumps the version, CHANGELOG heading, and badge. Flagging so it is not a surprise at merge time; the test slot t248 has a parallel collision risk for the same reason.

Everything else I re-checked is clean: the opencode roster carries steps: 40 with the prose renamed and no maxTurns leak, the .aidlc mirror keeps core shape, the docs matrix accurately describes both the supersede rule and per-harness enforcement, and the CHANGELOG entry is well-formed with an upgrade instruction. Fix the P1 (mechanical - restore one sentence, extend it to the cutoff branch, pin it) and decide a direction on the P2, and this is ready.

@jstrunk

jstrunk commented Jul 29, 2026

Copy link
Copy Markdown
Author

Re the request for a much higher turn limit: I posted field timing data on #612 (#612 (comment)) that sizes this cap, and it shows the two PRs really only work as a pair — I'd suggest deciding them together.

The short version: at reviewer effort medium there's a clean gap between normal and runaway reviews (p90 40 turns, max 60 across 132 observed reviews), so the 40–60 range is a real backstop. At inherited session effort there is no such gap — legitimate reviews reached 116 turns (p90 112), so 40 truncates a third of them and a "safe" ~120 cap protects against essentially nothing. If the review tier proposed on #612 lands, the cap in this PR should be 40–60; if it doesn't, maxTurns here buys little and I'd rather drop the cap than ship a meaningless number — though the §12a missing-verdict guard in this PR is worth keeping either way, since it catches silently incomplete reviews from any cause (crashes, opencode's forced text-only final step, model-routing bugs like anthropics/claude-code#81995), not just cap truncation.

@apackeer

apackeer commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thanks, Jeff. I want to make sure we evaluate these two PRs against the combined goal you described. Medium reviewer effort is the primary latency control: in your field data it reduced the architecture reviewer's median from 42.0 minutes to 4.1 minutes and p90 from 102.7 minutes to 14.6 minutes. The turn limit is then a deterministic backstop for outliers, while the incomplete-review guard makes a cap, crash, or harness cutoff fail clearly instead of leaving an undefined verdict path.

We closed #612 because changing the existing balanced tier to medium would override its inherit-session-effort contract. Your subsequent proposal for a dedicated review tier is materially different: balanced would continue to inherit session effort, while the two review-only agents would use medium effort. That appears to address the reason #612 was closed and deserves an explicit decision before we dispose of this PR.

@leandrodamascena, could we decide the following here?

  1. Do we want a dedicated review tier at medium effort?
  2. If so, what backstop should accompany it? The submitted value is 40 turns, while Jeff's later data identifies 40-60 as the meaningful range.
  3. Regardless of the tier decision, can we agree to carry forward the deterministic incomplete-review guard?

The current guard still needs to validate exactly one canonical verdict and record a terminal NOT-READY receipt for an incomplete attempt, but that is an implementation detail after the policy decision. Jeff, please do not rebase or revise this branch while we resolve that direction.

@apackeer apackeer mentioned this pull request Aug 10, 2026
4 tasks
@apackeer

Copy link
Copy Markdown
Contributor

@jstrunk Reviewer tier has now been implemented. i will rebase and merge this. @leandrodamascena your request changes remains, ill request rereview when ready to merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants