Skip to content

feat: bounded Build & Test → Code Generation failure loop-back (2.5.16) - #616

Open
jstrunk wants to merge 3 commits into
awslabs:v2from
jstrunk:feat/611-build-test-loopback
Open

feat: bounded Build & Test → Code Generation failure loop-back (2.5.16)#616
jstrunk wants to merge 3 commits into
awslabs:v2from
jstrunk:feat/611-build-test-loopback

Conversation

@jstrunk

@jstrunk jstrunk commented Jul 20, 2026

Copy link
Copy Markdown

Closes #611.

What

Gives Build & Test a sanctioned, bounded path back to Code Generation when the root cause of a failure lies in the generated code or an approach chosen there — replacing today's dead end ("2 attempts, then log and present the failed gate").

The 4-rung failure-escalation ladder (build-and-test.md Step 10):

  1. In-stage fix (max 2 attempts) — existing behavior, for root causes in this stage's own remit.
  2. Classify and price — root cause in generated code / a code-generation approach choice? Find a fix in a swappable dimension (newer image, driver, wheel index, a flag) and PRICE it (effort/cost/risk). Never declare a feasible path out of scope on an UNPRICED effort assumption.
  3. Autonomous bounded loop-back — under Construction Autonomy Mode: autonomous, with a priced fix and fewer than 3 entries in the ledger: record the diagnosis + plan, jump back to code-generation via the engine, apply the fix to the unit(s) the diagnosis names, replay forward. The failed run's gate is NOT presented; its learnings ritual defers to the passing run.
  4. Halt-and-ask — gated mode, bound exhausted, or no identifiable fix: a priced 3-option question (Retry with fix [price] / Accept failure / Abort). Giving up is the human's decision, never the agent's.

The counter is a crash-safe artifact ledger: an append-only ## Loop-Back Log in test-results.md — the entry count IS the bound (max 3 per intent). It survives the backward jump (jumps reset checkboxes, never artifacts), is colocated with the diagnosis, and is readable at the final gate; STAGE_JUMPED audit rows remain the deterministic cross-check. Human-directed jumps don't count.

Design decisions (vs. the field predecessor)

  • Engine-routed jump: the procedure runs aidlc-orchestrate.ts next --stage code-generation and executes the exact aidlc-jump.ts execute command the engine prints — never a hand-composed jump call.
  • Autonomy semantics: rung 3 keys off the existing Construction Autonomy Mode: autonomous grant; the replayed code-generation gate is auto-approved with an explicit --user-input "Autonomous loop-back N per stage-protocol §1" marker — a repair of the approved shape, not a new autonomy inference (checklist item 6 intact).
  • Swarm interaction (rewritten for 2.5.0's run-floor-stamped convergence ledger): on a loop-back replay that re-emits invoke-swarm, run check on every unit first and claim already-green units at finalize without a worker turn; dispatch workers only for units the planned fix targets or that fail the check.
  • Artifact Re-use under autonomy: the 3-option re-use question is not presented — deterministically Modify for targeted units, Keep for the rest, Modify for build-and-test itself (Redo forbidden — it would erase the ledger); every auto-decision still audited via reuse-artifact.
  • --single runs stop at rung 2 (no main-workflow position to move); priced options are logged and presented at that run's gate.

Changes

  • core/aidlc-common/stages/construction/build-and-test.md — Step 10 ladder + ## Loop-Back Log artifact shape.
  • core/aidlc-common/protocols/stage-protocol.md — new §1 subsection "Build-and-Test failure loop-back (3.6 → 3.5)" (purely additive sibling of the pinned Halt-and-ask block; t34/t76 green), carve-out sentence on the NO EMERGENT BEHAVIOR RULE, EXCEPTION sentence on checklist item 5, Artifact Re-use auto-decision rule.
  • core/aidlc-common/protocols/stage-protocol-recovery.md — crash-resume bullet (logged fix without a matching STAGE_JUMPED ⇒ re-execute the jump, don't re-diagnose; resume count = ledger rows).
  • harness/{claude,kiro,kiro-ide,codex,opencode}/skills/aidlc/SKILL.md — exception on the STAGE RITUAL IS ATOMIC principle (codex had no such bullet; it gains the full bullet with the exception).
  • New tests/integration/t245-build-test-loopback-prose.test.ts — 24 tests / 73 assertions pinning the ladder, §1 subsection, ledger paragraph, engine routing, swarm path, halt-and-ask template, recovery bullet, and the SKILL exception across authored + dist copies of all five harnesses.
  • Docs: docs/reference/04-stages/construction.md, 04-stage-protocol.md, 03-orchestrator.md.
  • Version 2.5.4 + CHANGELOG entry + README badge; all dists regenerated.

Testing

  • Smoke + unit tiers: 173 files, 4256 assertions, 0 failures.
  • New t245: 24 pass / 0 fail. Pinned-prose suites t34 / t76 / t44 / t87 / t68: 129 pass / 0 fail. Protocol readers t35–t37, t43, t47: 154 pass / 0 fail. Skill-parity/drift t181 / t55 / t239 / t129: 34 pass / 0 fail.
  • bun scripts/package.ts --check and the coverage-registry check both clean.

Notes for review

@apackeer

Copy link
Copy Markdown
Contributor

Review: bounded Build & Test -> Code Generation loop-back

Reviewed at head 579dd58. Since the PR is protocol prose + tests with no engine changes, every engine claim in the prose was verified against the tools it names, and they hold up well:

  • The engine-routed jump is real: next --stage code-generation resolves direction via aidlc-jump.ts resolve and prints exactly the execute --target ... --direction backward --scope ... command the prose describes (core/tools/aidlc-orchestrate.ts:2712-2714).
  • STAGE_JUMPED carries a literal Target: field (core/tools/aidlc-jump.ts:445-451), so the recovery bullet's "no matching STAGE_JUMPED (Target: code-generation)" check is greppable as written.
  • Backward jumps touch only the state file: handleExecute flips checkboxes and state fields and writes audit rows, with no artifact deletion anywhere in the file, so the Loop-Back Log genuinely survives the jump (core/tools/aidlc-jump.ts:313-332, :463).
  • The autonomous auto-approval clears the human-presence guard through the existing autonomy carve-out (core/tools/aidlc-state.ts:1784-1786), and Construction Autonomy Mode is a real, tool-read field (core/tools/aidlc-lib.ts:2658-2662), so rung 3 needs no new engine behavior.
  • The swarm run-floor story matches the code: SWARM_UNIT_CONVERGED rows are stamped with the stage's latest main-workflow STAGE_STARTED (core/tools/aidlc-swarm.ts:283-303, core/tools/aidlc-lib.ts:3451-3465), the jump mints a fresh STAGE_STARTED, and finalize re-verifies every claimed unit so the no-worker-turn claim path is safe at the tool level (core/tools/aidlc-swarm.ts:565-601).

Deterministic tiers reproduce the PR's numbers in my checkout: smoke+unit 173 files / 4256 assertions green, t245 24/24, the pinned-prose suites (t34/t76/t44/t87/t68) and protocol readers green, bun scripts/package.ts --check and the coverage-registry check clean, typecheck + biome clean.

Findings, in severity order.

Must-fix

1. Gated "Retry with fix" collides with the engine's settle-to-gate routing, so the fix is never applied on the replay. Because jumps preserve artifacts, every code-generation unit is still "covered" on disk after the backward jump. The off-swarm per-unit router therefore finds no uncovered unit and settles straight to a gate: true directive on the last unit (core/tools/aidlc-orchestrate.ts:2343-2361), and every harness SKILL instructs that gate: true on a per-unit stage means "every unit is already built, so run the learnings ritual and present the single approval gate" (for example harness/codex/skills/aidlc/SKILL.md:80) - no stage body, so the Artifact Re-use question never fires and the planned fix is never applied. The loop-back subsection's step 3 does say to apply the fix on re-entry, but the deterministic directive pushes the conductor the other way. Result if the conductor follows the directive: the gate is presented un-fixed, the human approves, build-and-test fails again, and each retry burns a ledger entry until the bound exhausts. The autonomous path has the same routing but its Artifact Re-use tail explicitly overrides it; the gated path has nothing. Needs either engine-side coverage invalidation on a loop-back jump, or explicit prose in the loop-back subsection stating that the re-entry directive arrives as gate: true and the planned fix must be applied (Modify on the targeted units) before that gate is presented - plus a test pinning whichever is chosen.

2. Version slot is stale and the PR is CONFLICTING. The body's assumption (companion PRs taking 2.5.1/2.5.2) is already broken: v2 is at 2.5.2 today (#595 merged as 2.5.1, #576 as 2.5.2). Per the CHANGELOG conflict policy this needs a rebase onto v2 plus a re-bump with the ## [2.5.4] heading renamed to match; t68 will catch a miss.

3. The PR falsifies a sentence it does not update. core/aidlc-common/protocols/stage-protocol.md:128 still says the Bolt halt-and-ask "is the one case where autonomous mode stops to consult the user." Rung 4 is now a second such case (autonomous run, bound exhausted or no identifiable fix). One-sentence fix; I checked that no test pins that sentence.

Should-fix

4. The no-fix branch cannot render the pinned template honestly. Rung 4 explicitly fires when "no identifiable fix exists" (core/aidlc-common/stages/construction/build-and-test.md:139-141), but the only halt-and-ask template has a mandatory Candidate fix: [fix] - estimated price: [...] slot and a "Retry with fix" option (core/aidlc-common/protocols/stage-protocol.md:214-224), and t245 pins the exact prompt string. A no-fix variant (Accept failure / Abort, or Retry-as-is) is needed, with matching t245 assertions.

5. --single promises a gate that isolated runs forbid. core/aidlc-common/stages/construction/build-and-test.md:149 says priced options are "presented at this run's gate", but the isolated stage-runner branch in every harness SKILL says "Do not ... present a workflow gate ... present the isolated-run summary and STOP" (harness/claude/skills/aidlc/SKILL.md:50 and the same bullet in the other four). Reword to "in the isolated-run summary"; the same phrase also appears at docs/reference/04-stages/construction.md:904 and in the CHANGELOG bullet. t245's single-run test pins "Stop at rung 2" but not the gate phrase, so the reword is test-safe.

6. The crash-resume bullet is in the wrong recovery section. It was appended under "Stage re-run" (core/aidlc-common/protocols/stage-protocol-recovery.md:119-125), which covers "user requested changes after approval". The scenario it describes (session died between logging and jumping) is detected on session resume, a different section (stage-protocol-recovery.md:44). A resuming conductor reading the resume path will not encounter it; move or cross-reference it under "Session resume". t245's recovery assertions are placement-agnostic.

7. The swarm cheap path hides two unenforced premises. (a) prepare hard-errors on replay if the prior attempt's worktrees or bolt-<slug> branches still exist (core/tools/aidlc-worktree.ts:212-220), and nothing in aidlc-swarm.ts ever discards them - a crash or halt-and-ask mid-swarm leaves them in place, so the first tool call of the prescribed replay can fail with an error the prose never mentions. (b) The premise "the prior run's code is in the base the worktrees forked from" is not guaranteed: swarm finalize merges only AIDLC metadata, never generated source (core/tools/aidlc-orchestrate.ts:805-812, core/tools/aidlc-bolt.ts:412-462); if the git code merge never ran on the failed attempt, every check is red and the cheap path silently degrades to full re-dispatch. A sentence handling the prepare collision (discard or adopt existing worktrees first) and softening the already-green premise would cover both.

Nits

  • The Artifact Re-use tail's abbreviated audit call (aidlc-state.ts reuse-artifact --decision <keep|modify>) omits the required positional slug and --artifacts flag, so run literally it errors (core/tools/aidlc-state.ts:2765-2772). Conductors execute this prose; worth spelling out the full form or pointing at the canonical block above it.
  • The commit message says t243; the file is t245. Also t244 is unclaimed on v2 and in the open PRs I checked, so the gap will confuse the next slot hunt - consider t244 or a note.
  • The max-3 bound, append-only rule, and Redo prohibition are all prose-only (nothing deterministic refuses a 4th backward jump or counts the ledger). Consistent with the PR's declared scope, but a cheap follow-up would be aidlc-jump.ts execute refusing a backward build-and-test -> code-generation jump when the ledger already holds 3 entries. Could ride the unit-scoped-redo follow-up already noted in the issue.

Happy to re-review after the rebase; the engine-claim verification above should carry over unchanged since no tool code moves.

fix):**

```question
prompt: "Build and Test failed: [short error]. Root cause: [diagnosis]. Candidate fix: [fix] — estimated price: [effort/cost/risk]. Loop-backs used: [N]/3. How would you like to proceed?"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UX: “price” normally implies a monetary value, but this prompt defines it as a combined estimate of effort, cost, and risk. Users may reasonably look for a dollar amount or misread what is being compared. Please rename this to estimated impact and expose the dimensions separately, for example: Estimated impact — effort: [...]; financial cost: [...]; risk: [...]. The related price / priced / unpriced wording and the Loop-Back Log’s Price field should be changed consistently.

jstrunk added 2 commits July 28, 2026 15:38
Implements awslabs#611. Build and Test's Step 10 "On failure" handling becomes a
4-rung failure-escalation ladder: (1) in-stage fix, max 2 attempts, for root
causes in the stage's own remit; (2) classify and price a candidate fix in a
swappable dimension - never write a feasible path off on an unpriced effort
assumption; (3) under Construction Autonomy Mode: autonomous, a bounded
autonomous loop-back (max 3 per intent) that records the diagnosis + priced
fix and replays through code-generation; (4) a priced 3-option halt-and-ask
(Retry with fix [price] / Accept failure / Abort).

- stage-protocol.md gains the sibling S1 subsection "Build-and-Test failure
  loop-back (3.6 -> 3.5)": the sanctioned carve-outs on the NO EMERGENT
  BEHAVIOR RULE and checklist item 5, the crash-safe append-only
  `## Loop-Back Log` artifact ledger in test-results.md (entry count IS the
  bound; STAGE_JUMPED rows stay the audit cross-check; human-directed jumps
  don't count), the ENGINE-routed jump procedure (next --stage
  code-generation emits the validated aidlc-jump.ts execute print), the
  Artifact Re-use auto-decision rule (Modify targeted units / Keep others /
  Modify build-and-test itself; Redo forbidden - it erases the ledger), the
  swarm check-first cheap path, and the priced halt-and-ask template.
- stage-protocol-recovery.md: crash-resume bullet - a logged planned fix with
  no matching STAGE_JUMPED (Target: code-generation) after it means the
  session died between logging and jumping; re-execute the jump, don't
  re-diagnose; resume count is the ledger's entry count.
- All five conductor SKILLs gain the parenthetical exception on the STAGE
  RITUAL IS ATOMIC bullet (codex gains the bullet itself, previously absent).
- Single-stage (--single) runs stop at rung 2: no main-workflow position to
  move; priced options are logged and presented at that run's gate.
- New prose-pin test tests/integration/t243-build-test-loopback-prose.test.ts
  (t34-style, mechanism none) pins the ladder, the S1 subsection + ledger
  paragraph, the recovery bullet, and the SKILL exception across authored +
  dist copies of every harness.
- Docs: construction reference chapter Build and Test section rewritten to
  the ladder; stage-protocol + orchestrator reference No Emergent Behavior
  mentions name the second carve-out.
- Version 2.5.4 + CHANGELOG entry + README badge; dist regenerated via
  bun scripts/package.ts (byte-parity --check green).
Addresses apackeer's review on PR awslabs#616 (a field engagement's build-and-test
failure loop-back), rebased onto v2 at 9f91454 and re-bumped 2.5.4 -> 2.5.16
to avoid colliding with sibling PRs rebasing in parallel.

Must-fix:
- A gated "Retry with fix" replay hits the same engine routing as the
  autonomous path: because a backward jump preserves artifacts, every
  code-generation unit is still "covered" on disk, so the re-entry `next`
  call answers with `gate: true` straight to the approval gate, skipping the
  stage body and the ordinary Artifact Re-use question. stage-protocol.md
  now spells out the same override for the gated case (a new "Gated failure
  loop-back" paragraph mirroring "Autonomous failure loop-back"): the planned
  fix (Modify on the targeted unit(s), via the Artifact Re-use mechanism)
  must be applied BEFORE that gate is presented.
- Rung 4 (autonomous run, bound exhausted or no identifiable fix) is now
  named as a second case, alongside the Bolt halt-and-ask, where `autonomous`
  mode stops to consult the user.

Should-fix:
- Added a no-fix halt-and-ask template variant (Accept failure / Abort, no
  "Candidate fix" slot, no "Retry with fix" option) for when rung 2 finds no
  identifiable fix; the priced 3-option template is used only when it did.
- Reworded "presented at this/that run's gate" to "in this/that run's
  isolated-run summary" (build-and-test.md, docs/reference/04-stages/
  construction.md, the CHANGELOG bullet) - isolated `--single` runs never
  present a workflow gate.
- Moved the crash-resume ("logged-but-not-jumped") detection bullet from
  stage-protocol-recovery.md's "Stage re-run" section (user-requested
  changes after approval - the wrong scenario) to "Session resume" (session
  died mid-jump); left a cross-reference behind.
- Swarm cheap-path prose now names the two unenforced premises: `prepare`
  hard-errors on a replay if a prior attempt's worktrees/branches still
  exist (discard or adopt them first), and the "already-green" claim holds
  only once the prior attempt's code merge actually completed (otherwise
  every `check` comes back red and the path degrades to full re-dispatch).

Nit:
- Spelled out the abbreviated `aidlc-state.ts reuse-artifact` audit call in
  full (positional slug + `--artifacts`) in both the autonomous and new
  gated paragraphs.

- Renamed tests/integration/t245-build-test-loopback-prose.test.ts to
  t250 (t245 collided with unrelated merged work) and added assertions
  pinning every fix above; the commit-message/file-number mismatch nit is
  resolved by this rename.
- dist regenerated via bun scripts/package.ts; byte-parity --check green.
@jstrunk
jstrunk force-pushed the feat/611-build-test-loopback branch from 579dd58 to fdf801a Compare July 28, 2026 15:52
@jstrunk jstrunk changed the title feat: bounded Build & Test → Code Generation failure loop-back (2.5.4) feat: bounded Build & Test → Code Generation failure loop-back (2.5.16) Jul 28, 2026
@jstrunk

jstrunk commented Jul 28, 2026

Copy link
Copy Markdown
Author

Thanks for the thorough review — addressed all of it. Summary below, plus the rebase.

Rebase + re-bump: rebased onto current v2 (9f91454). Bookkeeping conflicts resolved: version bumped 2.5.4 → 2.5.16 (pre-assigned to avoid colliding with sibling PRs rebasing in parallel), CHANGELOG heading/date and README badge updated to match. New commit: fdf801a.

Test rename: tests/integration/t245-build-test-loopback-prose.test.tst250-build-test-loopback-prose.test.ts (t245 was claimed by unrelated merged work in the interim; t250 is the pre-assigned collision-free slot). This also resolves the commit-message/file-number mismatch nit — the commit message and file now agree.

Must-fix #1 (gated "Retry with fix" collides with settle-to-gate routing): Confirmed the issue — a backward jump preserves artifacts, so every code-generation unit is still "covered" on disk, and the re-entry next call answers with gate: true straight to the approval gate, skipping the stage body (and the ordinary Artifact Re-use question) for the gated path exactly as you described. Went with the prose fix, mirroring the autonomous path's existing override: added a new "Gated failure loop-back" paragraph in stage-protocol.md's Artifact Re-use section, sibling to "Autonomous failure loop-back", stating the same deterministic Modify/Keep decisions must be applied via aidlc-state.ts reuse-artifact before that gate is presented. Cross-referenced from the halt-and-ask's "Retry with fix" description. Pinned in t250.

Must-fix #2 (version slot): Done — 2.5.16, per above.

Must-fix #3 (Bolt halt-and-ask "one case" sentence): Updated stage-protocol.md's Bolt halt-and-ask sentence to name rung 4 (bound exhausted / no identifiable fix, under autonomous) as a second case where autonomous mode stops to consult the user, alongside the Bolt halt-and-ask.

Should-fix #4 (no-fix halt-and-ask variant): Added a distinct no-fix template in stage-protocol.md — Accept failure / Abort only, no Candidate fix slot, no "Retry with fix" option (nothing to invent a fix to retry with). The stage body now picks the variant based on whether rung 2 actually produced a priced candidate fix. Pinned in t250.

Should-fix #5 ("presented at this run's gate" vs. isolated runs): Reworded to "in this/that run's isolated-run summary" in build-and-test.md, docs/reference/04-stages/construction.md, and the CHANGELOG bullet — matching the existing SKILL wording ("present the isolated-run summary and STOP"). Confirmed the phrase doesn't appear in any harness SKILL.md, so no changes needed there.

Should-fix #6 (crash-resume bullet in the wrong section): Moved the logged-but-not-jumped detection bullet from "Stage re-run" (wrong scenario — that's for user-requested changes after approval) to "Session resume" (the actual scenario — session died mid-jump). Left a short cross-reference in "Stage re-run" so a reader landing there isn't stranded.

Should-fix #7 (swarm cheap-path unenforced premises): Added both sentences to the swarm-interaction paragraph: (a) check for and discard/adopt a prior attempt's leftover worktrees/bolt-<slug> branches before calling prepare (which hard-errors on collision), and (b) softened the "already-green" claim — it holds only once the prior attempt's git code merge actually completed; otherwise every check comes back red and the path degrades gracefully to full re-dispatch rather than silently claiming unbuilt units.

Nit (abbreviated audit call): Spelled out aidlc-state.ts reuse-artifact <slug> --decision <keep|modify> --artifacts "<...>" in full in both the autonomous and new gated paragraphs.

Nit (t243/t245 mismatch): Resolved by the rename above.

Nit (unenforced bound/append-only/no-Redo rules): Left as-is per your note — consistent with the PR's declared scope.

Verification:

  • bun scripts/package.ts / --check: clean, no drift
  • bun tests/gen-coverage-registry.ts --check: OK
  • bun run check (package --check + typecheck + lint): clean
  • bash tests/run-tests.sh --smoke --unit: 178 files, 4396 assertions, 0 failures
  • t250-build-test-loopback-prose.test.ts (renamed + expanded with new pinning assertions): 36 pass
  • t68-version-changelog-sync.test.ts: 7 pass (2.5.16 verified in sync across aidlc-version.ts, CHANGELOG, README)

Force-pushed to feat/611-build-test-loopback at fdf801a. PR title updated to reflect 2.5.16.

@apackeer

Copy link
Copy Markdown
Contributor

Re-review at fdf801a

All three must-fix and all four should-fix items from the round-1 comment body are verified fixed at this head, and the deterministic gates reproduce green in my checkout. One round-1 item is still open, though: the inline review comment on the "price" wording was not addressed, so I'm holding approval on that one item (details at the end). Everything else is done.

Must-fix 1 (gated "Retry with fix" vs settle-to-gate routing) - fixed as asked, prose route. The new "Gated failure loop-back" paragraph sits as a sibling of the autonomous override in stage-protocol.md's Artifact Re-use tail, states the gate: true re-entry mechanism explicitly ("Do not accept that directive at face value"), and orders the fix BEFORE the gate; the halt-and-ask's "Retry with fix" prose cross-references it. I re-confirmed the underlying routing on the rebased head (the covered-units settle path still emits gate: true on the last unit, core/tools/aidlc-orchestrate.ts:2296-2312), so the prose now matches what the engine actually does on both paths. Pinned by two dedicated t250 tests.

Must-fix 2 (version slot) - fixed. 2.5.16 across aidlc-version.ts, CHANGELOG heading, README badge; t68 green; scripts/ci-changelog-guard.ts origin/v2 reports all 137 base entries preserved + 1 new. The pre-assignment also means this PR survives sibling merges below 2.5.16 without another re-bump, which is the right trade. One cosmetic consequence: 2.5.12-2.5.15 will read as gaps in the CHANGELOG until (unless) the sibling PRs claiming them land; nothing enforces against that.

Must-fix 3 ("the one case" sentence) - fixed; no "one case where" phrasing remains anywhere in core/docs/harness, and the replacement names rung 4 precisely (bound exhausted or no identifiable fix, under autonomous). Pinned.

Should-fix 4 (no-fix variant) - fixed. The no-fix template drops both the Candidate fix: slot and "Retry with fix", the chooser sentence keys on rung 2's actual output, and the t250 assertion isolates the question fence so the explanatory prose can still name the omitted option. Nice touch making the omission self-justifying ("a fabricated fix to retry with").

Should-fix 5 (isolated-run summary) - fixed; zero "at this/that run's gate" hits remain in core, docs, harness, or CHANGELOG, and the negative assertion in t250 guards regressions.

Should-fix 6 (crash-resume placement) - fixed; the detection bullet now lives under "Session resume" (where a resuming conductor actually reads) with a cross-reference left in "Stage re-run". t250 pins the ordering structurally.

Should-fix 7 (swarm premises) - fixed. The prepare-collision sentence is executable as written (aidlc-worktree.ts discard removes both the worktree and the bolt-<slug> branch, and prepare still hard-errors on either surviving), and the already-green premise is correctly scoped to "only once that attempt's git code merge actually completed" with the graceful-degradation outcome spelled out.

Nits - the audit call is spelled out in full in both override paragraphs (matches the tool's real usage line, aidlc-state.ts:3010); the t243/file mismatch is resolved by the t250 rename, and t250 is collision-free among the open PRs I checked (t248 is claimed twice, by #660 and #613 - not this PR's problem, but flagging it for whoever rebases second; t249 is #617's).

My gate runs at fdf801a (worktree checkout): smoke+unit 178 files / 4396 assertions green (matches your numbers exactly), t250 36/36, the pinned-prose suites (t34/t44/t76/t87) + t68 + t181 conductor-skill parity green, bun run check (package --check + both tsconfigs + biome) clean, coverage-registry --check clean.

Two small observations, neither blocking:

  1. The "Gated failure loop-back" paragraph names only Construction Autonomy Mode: gated, while rung 4's trigger is "gated (or unset)". The mode-agnostic cross-reference from "Retry with fix" routes an unset-mode conductor there anyway, so behavior is covered; widening the paragraph's opening to "gated (or unset)" would make it exact.
  2. The codex conductor SKILL gains the whole STAGE RITUAL IS ATOMIC bullet (it had none before this PR). That's the right call - t250's every-harness pin requires it and t181 parity stays green - just noting it's an alignment, not a carry-over.

Outstanding: the "price" -> "estimated impact" rename (round-1 inline comment)

The inline review comment on stage-protocol.md from round 1 asked to rename "price" (which reads as a monetary value) to estimated impact with the dimensions exposed separately (Estimated impact - effort: [...]; financial cost: [...]; risk: [...]), changing the price / priced / unpriced wording and the Loop-Back Log's Price field consistently. That comment was not addressed at this head: the priced-variant prompt still says estimated price: [effort/cost/risk] (stage-protocol.md:227), the ledger shape is still Diagnosis / Root-cause stage / Planned fix / Price (stage-protocol.md:178, build-and-test.md:116), rung 2 still says "PRICE it" (build-and-test.md:131), and the wording has since spread to docs/reference/04-stages/construction.md and the CHANGELOG entry, and is now pinned by t250 in 16 assertions. So the rename now touches four surfaces: the two protocol/stage files, the reference doc, the CHANGELOG bullet, and the matching t250 strings.

Holding approval on that one item - everything else is resolved, so it should be a quick mechanical sweep. Happy to approve as soon as it lands.

One process item: CI shows action_required on this head - as a fork PR it needs a maintainer's "Approve and run" click before the v2 gate actually executes; worth doing before merge so the required checks land.

apackeer added a commit that referenced this pull request Jul 29, 2026
Follow-up to the allowlist review. Three things:

Version slot: 2.5.16 was already claimed by #616 with an identical
"## [2.5.16] - 2026-07-28" heading, so this re-bumps to 2.5.17 per the
CHANGELOG conflict-trap rule (version file, README badge, heading).

t252 permission model: evaluate() refused any command containing a
separator or metacharacter before consulting the allowlist. That was
wrong in both directions. Live kiro-cli 2.12.1 RUNS a chain whose every
segment is allowlisted (`bun .kiro/tools/<t>.ts && date -u` executed
unprompted), so the model reported "ask" for something the binary
allows; and because the refusal was unconditional, six MUST_ASK entries
passed without exercising a shipped pattern at all - they would stay
green against an allowlist of `.*`, which is the failure mode this test
exists to prevent.

evaluate() now splits on `&&`/`;`/`|`/`&`/newline (quote-aware, and
newline included because Rust's negated classes match it) and requires
every segment to be allowed. Adds MUST_ALLOW_CHAINS for the live-verified
all-segments-allowed case, newline and background-operator cases to
MUST_ASK, and a meta-test asserting no MUST_ASK entry passes under a
wide-open allowlist. Verified by mutation: re-adding the removed
absolute-path pattern turns t252 red.

TAIL_METACHARACTERS narrows to `$(`, backtick, `<`, `>` - the forms live
2.12.1 actually gates. A bare `$` is excluded because
`bun $KIRO_PROJECT_DIR/.kiro/tools/<t>.ts` ran unprompted under a config
that allowlisted it, so expansion alone does not gate.

Rationale wording: the comment, doc note, and CHANGELOG said absolute
paths stay gated because a regex "cannot prove those forms still target
this project across supported Kiro releases". The real reason is that a
pattern checks a path's shape, not its trustworthiness - a grant for any
/.../.kiro/tools/*.ts also pre-approves a script planted in a
world-writable directory, which is the verified finding. Also rewrites
the stale 2.5.16 summary paragraph, which still described the superseded
wider-allowlist approach.
apackeer added a commit that referenced this pull request Jul 30, 2026
Follow-up to the allowlist review. Three things:

Version slot: 2.5.16 was already claimed by #616 with an identical
"## [2.5.16] - 2026-07-28" heading, so this re-bumps to 2.5.17 per the
CHANGELOG conflict-trap rule (version file, README badge, heading).

t252 permission model: evaluate() refused any command containing a
separator or metacharacter before consulting the allowlist. That was
wrong in both directions. Live kiro-cli 2.12.1 RUNS a chain whose every
segment is allowlisted (`bun .kiro/tools/<t>.ts && date -u` executed
unprompted), so the model reported "ask" for something the binary
allows; and because the refusal was unconditional, six MUST_ASK entries
passed without exercising a shipped pattern at all - they would stay
green against an allowlist of `.*`, which is the failure mode this test
exists to prevent.

evaluate() now splits on `&&`/`;`/`|`/`&`/newline (quote-aware, and
newline included because Rust's negated classes match it) and requires
every segment to be allowed. Adds MUST_ALLOW_CHAINS for the live-verified
all-segments-allowed case, newline and background-operator cases to
MUST_ASK, and a meta-test asserting no MUST_ASK entry passes under a
wide-open allowlist. Verified by mutation: re-adding the removed
absolute-path pattern turns t252 red.

TAIL_METACHARACTERS narrows to `$(`, backtick, `<`, `>` - the forms live
2.12.1 actually gates. A bare `$` is excluded because
`bun $KIRO_PROJECT_DIR/.kiro/tools/<t>.ts` ran unprompted under a config
that allowlisted it, so expansion alone does not gate.

Rationale wording: the comment, doc note, and CHANGELOG said absolute
paths stay gated because a regex "cannot prove those forms still target
this project across supported Kiro releases". The real reason is that a
pattern checks a path's shape, not its trustworthiness - a grant for any
/.../.kiro/tools/*.ts also pre-approves a script planted in a
world-writable directory, which is the verified finding. Also rewrites
the stale 2.5.16 summary paragraph, which still described the superseded
wider-allowlist approach.
apackeer added a commit that referenced this pull request Jul 30, 2026
* fix(kiro): harden the execute_bash permission lists (2.5.16)

Kiro wraps every `toolsSettings.execute_bash` pattern as `\A<pat>\z` (upstream
crates/chat-cli/src/cli/chat/tools/execute/mod.rs:130), so matching is
full-string, not prefix. The shipped patterns were written as if they were
prefixes, which made them simultaneously too narrow and, in one place, too
broad. Verified live against kiro-cli 2.12.1.

Too narrow (each of these needed an interactive approval, and a session with no
approver -- `--no-interactive`, or an ACP client that ignores
`session/request_permission` -- refuses them outright and stalls the workflow):

  - `bun run .kiro/tools/<tool>.ts`
  - `bun ".kiro/tools/<tool>.ts"` (quoted path)
  - `bun /abs/path/to/project/.kiro/tools/<tool>.ts`
  - `cd <dir> && bun .kiro/tools/<tool>.ts`, the form a conductor reaches for
    when the session cwd is not the project root
  - a bare `date -u`, which the stage prose instructs but `date -u .*` could
    never match

Too broad: `bun \.kiro/tools/.*` let the trailing wildcard swallow path
traversal. `bun .kiro/tools/../../outside-tool.ts` executed unprompted, i.e.
any file on the machine was reachable through the pre-approved prefix.
Approved script paths are now a single filename (`[A-Za-z0-9._-]+\.ts`, no
slash in the class), so `../` cannot appear. `.kiro/tools/` is flat in every
dist tree, so no real tool call is lost.

Also fixed:

  - The Kiro IDE conductor's `KIRO_PROJECT_DIR` pattern had unescaped braces
    (`\${?...}?`), an invalid regex that upstream silently drops via
    `.filter(Result::is_ok)`. It was inert: that command form was never
    actually pre-approved. Braces are now escaped.
  - The 14 delegated personas carried a narrower list than the conductor (no
    KIRO_PROJECT_DIR, absolute-path, or cd forms) and could be refused
    mid-stage. They now share the conductor's shell surface on both harnesses.
  - `deniedCommands` was anchored too, so `rm -rf /.*` missed `rm -rf ~/x`,
    `rm -rf *`, and `rm -fr <path>`, and `git push .*` missed a bare
    `git push`. Broadened to catch all of them.

Allowing a bare `cd <path>` is safe because 2.12.1 evaluates each `&&`/`;`/`|`
segment separately: `cd /tmp && curl example.com` is still refused on the curl
segment (live-verified, as is `cd /tmp && rm -f ...`).

t252 asserts this behaviourally -- it re-implements Kiro's matcher and runs
real command strings through it, rather than pinning literal regex text, since
a literal-text assertion cannot distinguish a working pattern from an inert
one (exactly how the IDE pattern above shipped dead). Its validity check
models the Rust regex crate's stricter brace handling, because JS RegExp
accepts `{?` as a literal and would call the broken pattern valid. t148 grows
a narrower guard against the traversal wildcard returning.

* fix(kiro): address allowlist review findings

* fix(kiro): model chain segmentation in t252, re-bump to 2.5.17

Follow-up to the allowlist review. Three things:

Version slot: 2.5.16 was already claimed by #616 with an identical
"## [2.5.16] - 2026-07-28" heading, so this re-bumps to 2.5.17 per the
CHANGELOG conflict-trap rule (version file, README badge, heading).

t252 permission model: evaluate() refused any command containing a
separator or metacharacter before consulting the allowlist. That was
wrong in both directions. Live kiro-cli 2.12.1 RUNS a chain whose every
segment is allowlisted (`bun .kiro/tools/<t>.ts && date -u` executed
unprompted), so the model reported "ask" for something the binary
allows; and because the refusal was unconditional, six MUST_ASK entries
passed without exercising a shipped pattern at all - they would stay
green against an allowlist of `.*`, which is the failure mode this test
exists to prevent.

evaluate() now splits on `&&`/`;`/`|`/`&`/newline (quote-aware, and
newline included because Rust's negated classes match it) and requires
every segment to be allowed. Adds MUST_ALLOW_CHAINS for the live-verified
all-segments-allowed case, newline and background-operator cases to
MUST_ASK, and a meta-test asserting no MUST_ASK entry passes under a
wide-open allowlist. Verified by mutation: re-adding the removed
absolute-path pattern turns t252 red.

TAIL_METACHARACTERS narrows to `$(`, backtick, `<`, `>` - the forms live
2.12.1 actually gates. A bare `$` is excluded because
`bun $KIRO_PROJECT_DIR/.kiro/tools/<t>.ts` ran unprompted under a config
that allowlisted it, so expansion alone does not gate.

Rationale wording: the comment, doc note, and CHANGELOG said absolute
paths stay gated because a regex "cannot prove those forms still target
this project across supported Kiro releases". The real reason is that a
pattern checks a path's shape, not its trustworthiness - a grant for any
/.../.kiro/tools/*.ts also pre-approves a script planted in a
world-writable directory, which is the verified finding. Also rewrites
the stale 2.5.16 summary paragraph, which still described the superseded
wider-allowlist approach.
@apackeer

apackeer commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thanks, Jeff. The previous re-review confirmed that you addressed all of the substantive findings from that pass. The one requested change still open is the terminology sweep from price to estimated impact, with effort, financial cost, and risk shown separately.

@leandrodamascena, could you take the second review pass on this PR? There is one interaction I would particularly like independently verified before final approval: a backward jump invalidates the existing per-unit Code Generation review receipts, while preserved artifacts make the replay route directly to the all-covered settle gate. Does the current replay protocol reliably produce fresh review receipts for every applicable unit before approval?

This is a verification question, not a new redesign request for Jeff. If the existing protocol already covers it, the terminology rename remains the only contributor change. If it exposes a real gap, we should identify the narrowest fix and handle the current-v2 integration as maintainers. Jeff, please do not spend time rebasing this branch while we complete that check.

Addresses the one outstanding review item on awslabs#616 (apackeer's round-1
inline comment on stage-protocol.md:227). "price" reads as a monetary
value while the loop-back defines it as a combined estimate of effort,
cost, and risk. Rename to "estimated impact" and expose the dimensions
separately in the halt-and-ask prompt:

  Candidate fix: [fix] — estimated impact — effort: [effort];
  financial cost: [cost]; risk: [risk].

Consistently changed across every surface the wording reached:
  - core/aidlc-common/protocols/stage-protocol.md (priced/no-fix
    halt-and-ask variants, ledger Estimated-impact field, autonomous
    procedure, give-up-violation sentence)
  - core/aidlc-common/stages/construction/build-and-test.md (rung 2
    "Classify and estimate impact", rungs 3/4, single-stage carve-out,
    Loop-Back Log field)
  - docs/reference/03-orchestrator.md, 04-stage-protocol.md,
    04-stages/construction.md
  - CHANGELOG.md 2.5.16 entry
  - tests/integration/t250-build-test-loopback-prose.test.ts (16 pinned
    assertions updated to the new wording)
  - dist/<harness>/ regenerated for all five harnesses

Prose-only refinement of the already-listed 2.5.16 entry; no version
bump. Unrelated aws-pricing / pricing-model / price-performance text
left untouched.

Verification: bun run check (package --check + typecheck + biome) clean;
bun tests/gen-coverage-registry.ts --check fresh; t250 36/36; t68 7/7;
smoke + unit tiers 178 files / 4396 assertions / 0 failures.
@jstrunk

jstrunk commented Aug 12, 2026

Copy link
Copy Markdown
Author

Addressed the one outstanding item — the priceestimated impact terminology rename — at 9d532d8, a commit on top of fdf801a (no rebase, per your request to leave the branch in place).

The halt-and-ask prompt now exposes the three dimensions separately, in the form you specified:

Candidate fix: [fix] — estimated impact — effort: [effort]; financial cost: [cost]; risk: [risk].

and the "Retry with fix" option description carries the same expanded form.

The price / priced / unpriced wording and the Loop-Back Log's Price field were changed consistently across every surface the terminology reached:

  • core/aidlc-common/protocols/stage-protocol.md — the halt-and-ask variants (now "impact-estimated variant" / no-fix variant), the ledger Estimated impact field, the autonomous procedure, and the give-up-violation sentence.
  • core/aidlc-common/stages/construction/build-and-test.md — rung 2 is now Classify and estimate impact (ESTIMATE ITS IMPACT — effort, financial cost, risk), rungs 3/4, the Loop-Back Log field, and the single-stage carve-out.
  • docs/reference/03-orchestrator.md, docs/reference/04-stage-protocol.md, docs/reference/04-stages/construction.md.
  • The CHANGELOG.md 2.5.16 entry.
  • tests/integration/t250-build-test-loopback-prose.test.ts — the 16 pinned assertions updated to the new wording.
  • dist/<harness>/ regenerated for all five harnesses.

Unrelated aws-pricing / pricing-model / price-performance text was left untouched. This is a prose-only refinement of the already-listed 2.5.16 entry, so no version bump (t68 stays green).

Verification:

  • bun run check (package --check + typecheck + biome) — clean.
  • bun scripts/package.ts --check — no drift across all five harness trees.
  • bun tests/gen-coverage-registry.ts --check — fresh; ratchet held.
  • bash tests/run-tests.sh --smoke --unit — 178 files, 4396 assertions, 0 failures.
  • Targeted: t250-build-test-loopback-prose.test.ts 36/36, t68-version-changelog-sync.test.ts 7/7.

No functional change; only the terminology. Happy to fold this into the maintainer-led v2 integration whenever that lands.

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 9d532d8. The requested “price” to “estimated impact” terminology change is complete and consistent.

[P1] Recreate every per-unit review receipt before the replayed settle gate

A backward jump emits STAGE_JUMPED, invalidating all prior Code Generation review receipts. Because artifacts remain covered, replay routes directly to a settle directive naming only the final Unit. The loop-back override applies the fix and then presents or auto-approves the gate without requiring fresh §12a reviews for every applicable Unit. The engine consequently refuses completion. Require fresh reviews for every applicable Unit after all fixes/reuse decisions and before the settle gate, with a behavioral receipt test.

Relevant locations: core/aidlc-common/protocols/stage-protocol.md:189-196, :260-267, and :1282-1295.

[P2] The swarm check-first path cannot finalize already-green Units without fresh reviews

prepare creates a new BOLT_STARTED boundary, and finalize requires a terminal REVIEW_COMPLETED after that boundary. The proposed cheap path claims already-green Units without a worker turn but never instructs a fresh reviewer dispatch, so finalize rejects those claims. Require a reviewer pass in each claimed Unit’s worktree after check; this can remain “without a builder turn,” but not without review.

Relevant locations: core/aidlc-common/protocols/stage-protocol.md:201-219 and core/tools/aidlc-swarm.ts:273-330.

Verification:

  • t250-build-test-loopback-prose: 36/36 passed
  • t205-gate-revision-backstop: 14/14 passed
  • bun scripts/package.ts --check: passed
  • git diff --check: passed

The branch conflicts with current v2; these fixes should be included in the maintainer-led integration rather than requiring another contributor rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants