From 7bf72c7326fe4a6c2211c4bc9832e76076085078 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 05:45:07 +0000 Subject: [PATCH 1/3] docs: design note applying AFlow-style MCTS optimization to the SDLC loop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maps AFlow (Zhang et al. 2025) onto autonomous-sdlc: what its MCTS workflow search assumes, why a naive port fails, and a three-tier adaptation — score every run from the existing state.json trace, reify the loop's config surface as a hashable workflow genome, then search genome variants with MCTS-lite using autoloop as the benchmark-mode evaluator harness. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01UT9Qjmu44qeUVEbwfbSfMk --- docs/aflow-sdlc-optimization.md | 238 ++++++++++++++++++++++++++++++++ 1 file changed, 238 insertions(+) create mode 100644 docs/aflow-sdlc-optimization.md diff --git a/docs/aflow-sdlc-optimization.md b/docs/aflow-sdlc-optimization.md new file mode 100644 index 0000000..3026953 --- /dev/null +++ b/docs/aflow-sdlc-optimization.md @@ -0,0 +1,238 @@ + + + +# AFlow for the SDLC Loop: Scored, Searched Workflow Evolution + +**Status**: Proposed design (not yet implemented) +**Target**: autonomous-sdlc (post-v2.1), composing with autoloop +**Companion**: `docs/sdlc-loop-redesign.md` (the v2 loop this optimizes) + +## 1. What AFlow Does (and What It Quietly Assumes) + +AFlow (Zhang et al., ICLR 2025) treats an agentic workflow as a **code-represented +graph** — nodes are LLM-invoking actions, edges are plain code — and searches the space +of workflows with MCTS: + +1. Initialize the tree with a template workflow. +2. **Select** a node using a soft mixture of score and uniform exploration. +3. **Expand**: an LLM proposes a modified workflow *conditioned on the parent's + evaluation performance* (its failure log). +4. **Execute and evaluate** the new workflow against a benchmark. +5. Keep the child only if it improves within a budget of rounds. +6. Stop when the top-k average score plateaus or the budget is exhausted. + +The loop is simple; the preconditions are not. AFlow works because it has: + +- **Cheap, repeatable evaluation** — hundreds of executions against fixed benchmark + problems (HumanEval, GSM8K) with **ground truth**, so a score is a pass rate, not a + judgment call. +- **A machine-manipulable workflow representation** — the LLM edits a small Python + graph drawn from a *constrained operator set* (Ensemble, Review, Revise…), not + arbitrary prose. +- **Low-noise scalar scores** — averaging over many problems per evaluation. + +Any honest port to the SDLC loop has to reconstruct or substitute each of these. + +## 2. Why the Naive Port Fails + +| AFlow assumes | The SDLC loop has | +|---|---| +| Evaluation = seconds, repeatable, hundreds of runs | Evaluation = one full feature build: hours of wall-clock, real tokens, non-repeatable task | +| Workflow = small code graph the LLM can rewrite | Workflow = a prose `SKILL.md` dispatch table + agents + hooks; free-editing it is unreviewable — and the loop **auto-approves its own actions**, so a self-edited workflow is a self-modifying autonomous system | +| Ground-truth score per problem | "Was this feature built well?" has no oracle; the PR review happens after the loop exits | +| Noise averaged over a benchmark | n=1 per (task, workflow) pair; tasks are not i.i.d. | + +So: no MCTS over raw `SKILL.md` edits, evaluated by vibes, one real feature at a time. +That version burns money and converges on nothing. + +## 3. What Already Maps: The Plugin Is Half an AFlow + +The interesting discovery is how much of AFlow's machinery the v2 loop already ships — +unscored and unsearched: + +| AFlow component | Existing counterpart | What's missing | +|---|---|---| +| Workflow representation | The loop's **config surface**: budgets, review-gate config (`reviewers`/`mode`), signs, dispatch-table behavior knobs | Not reified — scattered across `state.json`, `signs.md`, and prose; unversioned | +| Execute + trace | Every real `/sdlc` run writes a **complete execution trace**: `state.json` `history` (every transition, timestamped, with reason), `attempts`, `wait_ticks`, `decisions.jsonl`, `progress.md` | The trace is **discarded** at DONE; nothing computes a score from it | +| Expand (LLM proposes a modification conditioned on the failure log) | **`signs.md` + the feedback skill**: after watching the loop err, a guardrail line is added and replayed every iteration; `feedback consolidate` graduates durable ones | No score, no selection, no rollback — signs accumulate **monotonically whether or not they help**. This is expansion without evaluation | +| Evaluator harness | **autoloop**'s seven components: mutable artifact, scalar metric, immutable runner, quality gates, git checkpoint/rollback, results ledger | Never pointed at the SDLC workflow itself | +| Benchmark problems | `evals/` fixture datasets (skill-trigger evals) | No end-to-end SDLC task suite | + +AFlow's contribution, seen from here, is exactly the four missing pieces: **a score, a +ledger, a selection rule, and a tree of variants instead of a monotone pile of signs.** + +## 4. Design: Three Tiers + +Each tier is independently useful; each later tier consumes the previous one. + +### Tier 1 — Score every run (free telemetry, no search) + +Extend `sdlc_state.py` with a `score` command that computes a scalar from the trace at +terminal states, and have the DONE/BLOCKED path append a **run record** to a durable +ledger before `.sdlc/` markers are cleaned: + +``` +~/.claude/autonomous-sdlc/runs.jsonl # cross-project, like feedback storage +{"at": "...", "repo": "...", "feature": "user-auth", "genome": "sha256:ab12…", + "outcome": "DONE", "score": 0.74, "iterations": 21, "tasks_closed": 6, + "rework": {"verify_bounces": 1, "review_roundtrips": 1, "repairs": 0}, + "attempts_exceeded": 0, "wait_ticks": 40, "pr": "https://…"} +``` + +**Score function** (v1, weights to be tuned once data exists): + +``` +score = 0.5 * outcome # DONE-with-PR = 1.0, BLOCKED = 0.0 + + 0.2 * efficiency # 1 / (1 + iterations_per_closed_task - baseline) + + 0.2 * (1 - rework_rate) # backward transitions / total transitions + + 0.1 * autonomy # 1 - (attempts_exceeded + escalations) / tasks +``` + +**Secondary metrics — tracked, never optimized** (autoloop's Goodhart guard): +wall-clock, `wait_ticks`, decision count, and (future) post-merge signal — human review +comments on the PR, reverts of loop commits. If a genome improves the score while +secondary metrics degrade, that is a red flag, not a win. + +Tier 1 alone already answers a question the plugin can't answer today: *did that sign +we added last month actually help?* + +### Tier 2 — Reify the workflow genome + +Make the config surface explicit and hashable. A **genome** is a single JSON document — +`.sdlc/workflow.json`, written at `init`, immutable for the lifetime of a run: + +```json +{ + "genome_version": 1, + "budgets": { "max_iterations": 50, "max_attempts_per_task": 3 }, + "review": { "reviewers": ["code-review"], "mode": "block", "roundtrip_budget": 2 }, + "policy": { + "parallel_builder_threshold": 3, + "simplify_before_ship": true, + "plan_granularity": "default", + "spec_depth": "default" + }, + "signs": ["Sign: don't assume the helper exists — check first"], + "overlays": { "BUILD": "", "VERIFY": "", "REVIEW": "" } +} +``` + +- The loop reads it in the iteration ritual; signs move here from `signs.md` (which + becomes the *inbox* for candidate signs, promoted into the genome by the optimizer). +- `overlays` are bounded prompt additions per state — the constrained analogue of + AFlow editing a node's prompt. +- Run records carry the genome's content hash, so the ledger links every score to the + exact workflow that produced it. + +This is AFlow's "workflow as code," with the same move AFlow itself makes: **mutation +is confined to a declared parameter space** (its operator set; our schema), not +arbitrary edits. The `SKILL.md`, agents, and hooks stay fixed — they are the *substrate*, +the genome is the searchable part. + +### Tier 3 — The optimizer: MCTS-lite over genomes + +A new `sdlc-optimize` skill (plus `/sdlc-optimize` command) owning a variant tree at +`~/.claude/autonomous-sdlc/optimize/tree.json`. Nodes are genomes with score statistics +`{n, mean, var}`; each edge records the modification that produced the child. + +One optimization round, mirroring AFlow steps 2–5: + +1. **Select** — soft mixture, as in the paper: with probability ε (default 0.2) pick a + uniform-random node; otherwise softmax over the top-k nodes' **pessimistic** scores + (`mean − stderr`, so an n=1 fluke doesn't dominate selection). +2. **Expand** — the LLM proposes **one** modification to the selected genome, + conditioned on that genome's *failure log*: the lowest-scoring run traces, excerpted + where points were lost (repeated REVIEW→BUILD bounces, `EXCEEDED` tasks, the + `BLOCKED` reason). Output must validate against the genome schema — an invalid + proposal is rejected and retried, never patched around. +3. **Evaluate** — two modes: + - **Shadow mode** (slow, free, noisy): the child genome becomes the active genome + for the next K real `/sdlc` runs; Tier 1 scores accrue to it. + - **Benchmark mode** (repeatable, costs tokens): headless runs over a fixture suite + — see below. +4. **Keep/backprop** — add the child to the tree only if its mean over its evaluation + budget beats the parent's; otherwise record the failed edge (so expansion doesn't + re-propose it) and discard. +5. **Stop** — top-k average plateaus for R consecutive rounds (default 3) or the round + budget (default 10) is exhausted. The winner is offered to the user as the new + default genome — and durable signs graduate to `feedback save`, exactly the + existing consolidation path, now score-backed. + +### Benchmark mode composes with autoloop + +Benchmark mode is not new machinery — it is **an autoloop whose mutable artifact is the +genome**: + +| autoloop component | Instantiation | +|---|---| +| Mutable artifact | the candidate genome JSON | +| Immutable context | `evals/sdlc-bench/` fixture repos + task specs | +| Runner (`auto/run.sh`) | for each bench task: clone fixture to scratch, `claude -p "/sdlc ''"` headless with the candidate genome and small budgets (`max_iterations` ≈ 15), then run the fixture's **held-out acceptance tests** against the produced branch, then `sdlc_state.py score` | +| Primary metric | mean composite: held-out pass rate × trace score | +| Secondary metrics | wall-clock, iterations, token spend | +| Quality gates | genome schema validation (hard fail) before any run | +| Checkpoint/rollback | git commit the genome on improvement, reset on regression | +| Ledger | `results.tsv` — one row per (genome, task) | + +The held-out test suite is the important recovery: fixture tasks ship acceptance tests +the loop never sees, which restores AFlow's **ground truth** property. A 3–5 task suite +(small Python repos: add a feature, fix a bug with a regression test, refactor under +tests) lives in `evals/sdlc-bench/` alongside the existing eval fixtures. + +## 5. Cost and Noise, Honestly + +- **Benchmark runs are expensive.** One bench task ≈ one small headless loop — minutes + and real tokens. AFlow-scale search (hundreds of evaluations) is out. Defaults are + sized accordingly: 10 rounds × 3 tasks × 1 seed ≈ 30 loop runs per optimization + session, overnight-shaped — exactly autoloop's operating envelope. Baseline scores + are computed once and reused. +- **Shadow-mode scores are noisy and non-i.i.d.** Real features differ wildly in + difficulty. Mitigations: pessimistic scoring (above), a minimum n before a child can + displace its parent, and preferring genome deltas that plausibly generalize (budget + and gate changes) over task-shaped ones. +- **Sequence the tiers.** Tier 1 costs almost nothing and creates the dataset. Do not + build Tier 3 before Tier 1 has accumulated enough real-run records to sanity-check + the score function against human judgment of "that run went well." + +## 6. What We Deliberately Do NOT Do + +- **No self-editing of `SKILL.md`, agents, or hooks by the optimizer.** Mutation is + confined to the genome schema. Graduating a proven overlay into the skill text + remains a human-reviewed PR — the optimizer produces the *evidence*, not the commit. + (A system that auto-approves its own actions must not also choose its own rules.) +- **No mid-run mutation.** A genome is fixed for a run's lifetime; scores attribute + cleanly. +- **No optimizing secondary metrics**, ever. They exist to veto, not to climb. +- **Signs are not deleted as a concept** — they become genome content with scores + attached, which finally makes bad signs *removable*. Today's append-only `signs.md` + is expansion without evaluation; that is the one part of the current design this + proposal retires. + +## 7. New Pieces and Migration + +``` +plugins/autonomous-sdlc/ +├── scripts/ +│ └── sdlc_state.py # + `score` command, + run-record append at DONE/BLOCKED +├── skills/ +│ └── sdlc-optimize/ # NEW (Tier 3): tree ops, select/expand/evaluate/keep +│ └── SKILL.md +├── commands/ +│ └── sdlc-optimize.md # NEW: kick off / resume an optimization session +evals/ +└── sdlc-bench/ # NEW (Tier 3): fixture repos + task specs + held-out tests +``` + +1. **Tier 1** — `score` command + run ledger + genome extraction of the knobs that + already exist (budgets, review config, signs). No behavior change; pure telemetry. +2. **Tier 2** — loop reads `.sdlc/workflow.json`; `signs.md` becomes the inbox. +3. **Tier 3 (benchmark)** — `evals/sdlc-bench/` suite + autoloop-generated runner. +4. **Tier 3 (search)** — the `sdlc-optimize` skill; dogfood one overnight session on + the bench suite before touching shadow mode. + +## Sources + +- [AFlow: Automating Agentic Workflow Generation (Zhang et al., ICLR 2025)](https://arxiv.org/abs/2410.10762) +- `docs/sdlc-loop-redesign.md` — the loop being optimized, and its signs/feedback mechanism +- `plugins/autoloop/skills/autoloop/SKILL.md` — the seven-component evaluator harness reused by benchmark mode +- [Anthropic Engineering — Effective harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) From 67ef2a6ce658a36f227e475366cea67bcc8c5d1f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 05:58:35 +0000 Subject: [PATCH 2/3] docs: recenter AFlow note on capture-every-run + periodic retro flow Per discussion: instead of benchmark-driven MCTS as the primary mechanism, every real /sdlc run archives its trace (state history, decisions, signs, escalations) to a durable user-level ledger with a computed score, and a periodic /sdlc-retro mines the accumulated records to propose skill improvements as a human-reviewed PR. Each retro first grades the previous one by comparing score windows across plugin versions (AFlow's keep-if-improved, amortized over real usage). Deliberate genome/benchmark search moves to future work. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01UT9Qjmu44qeUVEbwfbSfMk --- docs/aflow-sdlc-optimization.md | 405 +++++++++++++++----------------- 1 file changed, 195 insertions(+), 210 deletions(-) diff --git a/docs/aflow-sdlc-optimization.md b/docs/aflow-sdlc-optimization.md index 3026953..000c5cc 100644 --- a/docs/aflow-sdlc-optimization.md +++ b/docs/aflow-sdlc-optimization.md @@ -1,85 +1,105 @@ - - + + -# AFlow for the SDLC Loop: Scored, Searched Workflow Evolution +# Learning From Every Run: Feeding SDLC Traces Back Into the Skill **Status**: Proposed design (not yet implemented) -**Target**: autonomous-sdlc (post-v2.1), composing with autoloop -**Companion**: `docs/sdlc-loop-redesign.md` (the v2 loop this optimizes) +**Target**: autonomous-sdlc (post-v2.1) +**Companion**: `docs/sdlc-loop-redesign.md` (the loop being improved), +`docs/compound-engineering-roadmap.md` (this is the "compound" step applied to the loop itself) + +> **Revision note.** The first draft of this note proposed benchmark-driven MCTS over +> workflow variants as the primary mechanism (per AFlow, Zhang et al., ICLR 2025). +> Discussion inverted that: the primary mechanism is now **capturing every real run and +> periodically feeding the accumulated results back into the skill**. Deliberate search +> survives as future work (§8). AFlow still supplies the conceptual skeleton — score, +> ledger, LLM-proposed modification conditioned on failure logs, keep-only-if-improved — +> but evaluation is amortized over normal usage instead of purchased with benchmark runs. + +## 1. The Problem: The Loop Learns Nothing From Itself + +Every `/sdlc` run produces a complete execution trace and then throws it away: + +- `state.json` `history` — every transition, timestamped, with a reason. The rework + story (VERIFY→BUILD bounces, REVIEW round-trips, REPAIR entries) is right there. +- `attempts`, `wait_ticks`, `iteration` — where effort actually went. +- `decisions.jsonl` — every autonomous judgment call (kept in the PR body, but never + aggregated across runs). +- `progress.md`, `signs.md`, `escalation.md` — the narrative, the guardrails in force, + and why the loop gave up when it did. + +The DONE state "cleans `.sdlc/` markers"; BLOCKED runs — the most informative ones — +leave their trace stranded on an abandoned branch. Meanwhile the only improvement +mechanism, `signs.md` + the feedback skill, is **append-only expansion with no +evaluation**: a sign added after one bad iteration is replayed forever, whether or not +it ever helped, and nothing ever removes one. + +In AFlow's terms: the plugin already has *execution* and *expansion*, but no *score*, +no *ledger*, no *selection*, and no way to check whether a modification improved +anything. + +## 2. The Flow -## 1. What AFlow Does (and What It Quietly Assumes) - -AFlow (Zhang et al., ICLR 2025) treats an agentic workflow as a **code-represented -graph** — nodes are LLM-invoking actions, edges are plain code — and searches the space -of workflows with MCTS: - -1. Initialize the tree with a template workflow. -2. **Select** a node using a soft mixture of score and uniform exploration. -3. **Expand**: an LLM proposes a modified workflow *conditioned on the parent's - evaluation performance* (its failure log). -4. **Execute and evaluate** the new workflow against a benchmark. -5. Keep the child only if it improves within a budget of rounds. -6. Stop when the top-k average score plateaus or the budget is exhausted. - -The loop is simple; the preconditions are not. AFlow works because it has: - -- **Cheap, repeatable evaluation** — hundreds of executions against fixed benchmark - problems (HumanEval, GSM8K) with **ground truth**, so a score is a pass rate, not a - judgment call. -- **A machine-manipulable workflow representation** — the LLM edits a small Python - graph drawn from a *constrained operator set* (Ensemble, Review, Revise…), not - arbitrary prose. -- **Low-noise scalar scores** — averaging over many problems per evaluation. - -Any honest port to the SDLC loop has to reconstruct or substitute each of these. - -## 2. Why the Naive Port Fails - -| AFlow assumes | The SDLC loop has | -|---|---| -| Evaluation = seconds, repeatable, hundreds of runs | Evaluation = one full feature build: hours of wall-clock, real tokens, non-repeatable task | -| Workflow = small code graph the LLM can rewrite | Workflow = a prose `SKILL.md` dispatch table + agents + hooks; free-editing it is unreviewable — and the loop **auto-approves its own actions**, so a self-edited workflow is a self-modifying autonomous system | -| Ground-truth score per problem | "Was this feature built well?" has no oracle; the PR review happens after the loop exits | -| Noise averaged over a benchmark | n=1 per (task, workflow) pair; tasks are not i.i.d. | +``` + every real /sdlc run periodically (human-invoked) + ┌──────────────────────────┐ ┌────────────────────────────────┐ + /sdlc ──────►│ run loop ──► DONE/BLOCKED│ │ /sdlc-retro │ + │ │ │ │ mine ledger since last retro │ + │ ▼ │ │ ──► ≤5 grounded proposals │ + │ archive trace + score ──┼──────────► │ ──► PR against the plugin │ + │ (runs ledger, durable) │ ledger │ (human reviews, merges) │ + └──────────────────────────┘ └───────────────┬────────────────┘ + ▲ │ + │ next period's runs evaluate │ + └──────────── the merged changes ◄─────────────┘ + (next retro checks: did it help?) +``` -So: no MCTS over raw `SKILL.md` edits, evaluated by vibes, one real feature at a time. -That version burns money and converges on nothing. +Four stages, each independently shippable: **capture** (§3), **score** (§4), +**retro** (§5), **measure** (§6). -## 3. What Already Maps: The Plugin Is Half an AFlow +## 3. Capture: Archive Every Run, Automatically -The interesting discovery is how much of AFlow's machinery the v2 loop already ships — -unscored and unsearched: +Hook the archive into the terminal transition — `sdlc_state.py` archives whenever the +state becomes `DONE` **or** `BLOCKED` (including budget/no-progress force-blocks, which +go through the same `block()` path). Not a dispatch-table instruction the model can +forget; a side effect of the state machine it already must call. -| AFlow component | Existing counterpart | What's missing | -|---|---|---| -| Workflow representation | The loop's **config surface**: budgets, review-gate config (`reviewers`/`mode`), signs, dispatch-table behavior knobs | Not reified — scattered across `state.json`, `signs.md`, and prose; unversioned | -| Execute + trace | Every real `/sdlc` run writes a **complete execution trace**: `state.json` `history` (every transition, timestamped, with reason), `attempts`, `wait_ticks`, `decisions.jsonl`, `progress.md` | The trace is **discarded** at DONE; nothing computes a score from it | -| Expand (LLM proposes a modification conditioned on the failure log) | **`signs.md` + the feedback skill**: after watching the loop err, a guardrail line is added and replayed every iteration; `feedback consolidate` graduates durable ones | No score, no selection, no rollback — signs accumulate **monotonically whether or not they help**. This is expansion without evaluation | -| Evaluator harness | **autoloop**'s seven components: mutable artifact, scalar metric, immutable runner, quality gates, git checkpoint/rollback, results ledger | Never pointed at the SDLC workflow itself | -| Benchmark problems | `evals/` fixture datasets (skill-trigger evals) | No end-to-end SDLC task suite | +**Where**: user-level, cross-project — learning should compound across repos, exactly +like the feedback skill's storage: -AFlow's contribution, seen from here, is exactly the four missing pieces: **a score, a -ledger, a selection rule, and a tree of variants instead of a monotone pile of signs.** +``` +~/.claude/autonomous-sdlc/ +├── runs.jsonl # one summary line per run (the ledger) +└── runs/2026-07-08T05-41_claude-plugins_user-auth/ + ├── state.json # full history, attempts, budgets, review config + ├── decisions.jsonl + ├── progress.md + ├── signs.md # the guardrails that were in force + └── escalation.md # when BLOCKED +``` -## 4. Design: Three Tiers +**Ledger line** (the retro's primary input; the directory is the drill-down): -Each tier is independently useful; each later tier consumes the previous one. +```json +{"at": "...", "repo": "claude-plugins", "feature": "user-auth", + "plugin_version": "2.1.0", "outcome": "DONE", "score": 0.74, + "iterations": 21, "tasks_closed": 6, "attempts_exceeded": 0, + "rework": {"verify_bounces": 1, "review_roundtrips": 1, "repairs": 0}, + "wait_ticks": 40, "decisions": 9, "signs_active": 3, + "blocked_reason": null, "pr": "https://…"} +``` -### Tier 1 — Score every run (free telemetry, no search) +`plugin_version` (read from the installed plugin's `plugin.json`) is the load-bearing +field: it is what lets a later retro attribute score changes to a skill change (§6). -Extend `sdlc_state.py` with a `score` command that computes a scalar from the trace at -terminal states, and have the DONE/BLOCKED path append a **run record** to a durable -ledger before `.sdlc/` markers are cleaned: +Everything stays local to the user's machine; traces contain project narrative +(`progress.md` may quote code), so nothing is transmitted anywhere — the retro reads +the archive in place. -``` -~/.claude/autonomous-sdlc/runs.jsonl # cross-project, like feedback storage -{"at": "...", "repo": "...", "feature": "user-auth", "genome": "sha256:ab12…", - "outcome": "DONE", "score": 0.74, "iterations": 21, "tasks_closed": 6, - "rework": {"verify_bounces": 1, "review_roundtrips": 1, "repairs": 0}, - "attempts_exceeded": 0, "wait_ticks": 40, "pr": "https://…"} -``` +## 4. Score: One Number Per Run -**Score function** (v1, weights to be tuned once data exists): +Computed at archive time from the trace (`sdlc_state.py score`, also runnable ad hoc): ``` score = 0.5 * outcome # DONE-with-PR = 1.0, BLOCKED = 0.0 @@ -88,151 +108,116 @@ score = 0.5 * outcome # DONE-with-PR = 1.0, BLOCKED = 0.0 + 0.1 * autonomy # 1 - (attempts_exceeded + escalations) / tasks ``` -**Secondary metrics — tracked, never optimized** (autoloop's Goodhart guard): -wall-clock, `wait_ticks`, decision count, and (future) post-merge signal — human review -comments on the PR, reverts of loop commits. If a genome improves the score while -secondary metrics degrade, that is a red flag, not a win. - -Tier 1 alone already answers a question the plugin can't answer today: *did that sign -we added last month actually help?* - -### Tier 2 — Reify the workflow genome - -Make the config surface explicit and hashable. A **genome** is a single JSON document — -`.sdlc/workflow.json`, written at `init`, immutable for the lifetime of a run: - -```json -{ - "genome_version": 1, - "budgets": { "max_iterations": 50, "max_attempts_per_task": 3 }, - "review": { "reviewers": ["code-review"], "mode": "block", "roundtrip_budget": 2 }, - "policy": { - "parallel_builder_threshold": 3, - "simplify_before_ship": true, - "plan_granularity": "default", - "spec_depth": "default" - }, - "signs": ["Sign: don't assume the helper exists — check first"], - "overlays": { "BUILD": "", "VERIFY": "", "REVIEW": "" } -} -``` - -- The loop reads it in the iteration ritual; signs move here from `signs.md` (which - becomes the *inbox* for candidate signs, promoted into the genome by the optimizer). -- `overlays` are bounded prompt additions per state — the constrained analogue of - AFlow editing a node's prompt. -- Run records carry the genome's content hash, so the ledger links every score to the - exact workflow that produced it. - -This is AFlow's "workflow as code," with the same move AFlow itself makes: **mutation -is confined to a declared parameter space** (its operator set; our schema), not -arbitrary edits. The `SKILL.md`, agents, and hooks stay fixed — they are the *substrate*, -the genome is the searchable part. - -### Tier 3 — The optimizer: MCTS-lite over genomes - -A new `sdlc-optimize` skill (plus `/sdlc-optimize` command) owning a variant tree at -`~/.claude/autonomous-sdlc/optimize/tree.json`. Nodes are genomes with score statistics -`{n, mean, var}`; each edge records the modification that produced the child. - -One optimization round, mirroring AFlow steps 2–5: - -1. **Select** — soft mixture, as in the paper: with probability ε (default 0.2) pick a - uniform-random node; otherwise softmax over the top-k nodes' **pessimistic** scores - (`mean − stderr`, so an n=1 fluke doesn't dominate selection). -2. **Expand** — the LLM proposes **one** modification to the selected genome, - conditioned on that genome's *failure log*: the lowest-scoring run traces, excerpted - where points were lost (repeated REVIEW→BUILD bounces, `EXCEEDED` tasks, the - `BLOCKED` reason). Output must validate against the genome schema — an invalid - proposal is rejected and retried, never patched around. -3. **Evaluate** — two modes: - - **Shadow mode** (slow, free, noisy): the child genome becomes the active genome - for the next K real `/sdlc` runs; Tier 1 scores accrue to it. - - **Benchmark mode** (repeatable, costs tokens): headless runs over a fixture suite - — see below. -4. **Keep/backprop** — add the child to the tree only if its mean over its evaluation - budget beats the parent's; otherwise record the failed edge (so expansion doesn't - re-propose it) and discard. -5. **Stop** — top-k average plateaus for R consecutive rounds (default 3) or the round - budget (default 10) is exhausted. The winner is offered to the user as the new - default genome — and durable signs graduate to `feedback save`, exactly the - existing consolidation path, now score-backed. - -### Benchmark mode composes with autoloop - -Benchmark mode is not new machinery — it is **an autoloop whose mutable artifact is the -genome**: - -| autoloop component | Instantiation | -|---|---| -| Mutable artifact | the candidate genome JSON | -| Immutable context | `evals/sdlc-bench/` fixture repos + task specs | -| Runner (`auto/run.sh`) | for each bench task: clone fixture to scratch, `claude -p "/sdlc ''"` headless with the candidate genome and small budgets (`max_iterations` ≈ 15), then run the fixture's **held-out acceptance tests** against the produced branch, then `sdlc_state.py score` | -| Primary metric | mean composite: held-out pass rate × trace score | -| Secondary metrics | wall-clock, iterations, token spend | -| Quality gates | genome schema validation (hard fail) before any run | -| Checkpoint/rollback | git commit the genome on improvement, reset on regression | -| Ledger | `results.tsv` — one row per (genome, task) | - -The held-out test suite is the important recovery: fixture tasks ship acceptance tests -the loop never sees, which restores AFlow's **ground truth** property. A 3–5 task suite -(small Python repos: add a feature, fix a bug with a regression test, refactor under -tests) lives in `evals/sdlc-bench/` alongside the existing eval fixtures. - -## 5. Cost and Noise, Honestly - -- **Benchmark runs are expensive.** One bench task ≈ one small headless loop — minutes - and real tokens. AFlow-scale search (hundreds of evaluations) is out. Defaults are - sized accordingly: 10 rounds × 3 tasks × 1 seed ≈ 30 loop runs per optimization - session, overnight-shaped — exactly autoloop's operating envelope. Baseline scores - are computed once and reused. -- **Shadow-mode scores are noisy and non-i.i.d.** Real features differ wildly in - difficulty. Mitigations: pessimistic scoring (above), a minimum n before a child can - displace its parent, and preferring genome deltas that plausibly generalize (budget - and gate changes) over task-shaped ones. -- **Sequence the tiers.** Tier 1 costs almost nothing and creates the dataset. Do not - build Tier 3 before Tier 1 has accumulated enough real-run records to sanity-check - the score function against human judgment of "that run went well." - -## 6. What We Deliberately Do NOT Do - -- **No self-editing of `SKILL.md`, agents, or hooks by the optimizer.** Mutation is - confined to the genome schema. Graduating a proven overlay into the skill text - remains a human-reviewed PR — the optimizer produces the *evidence*, not the commit. - (A system that auto-approves its own actions must not also choose its own rules.) -- **No mid-run mutation.** A genome is fixed for a run's lifetime; scores attribute - cleanly. -- **No optimizing secondary metrics**, ever. They exist to veto, not to climb. -- **Signs are not deleted as a concept** — they become genome content with scores - attached, which finally makes bad signs *removable*. Today's append-only `signs.md` - is expansion without evaluation; that is the one part of the current design this - proposal retires. - -## 7. New Pieces and Migration - -``` -plugins/autonomous-sdlc/ -├── scripts/ -│ └── sdlc_state.py # + `score` command, + run-record append at DONE/BLOCKED -├── skills/ -│ └── sdlc-optimize/ # NEW (Tier 3): tree ops, select/expand/evaluate/keep -│ └── SKILL.md -├── commands/ -│ └── sdlc-optimize.md # NEW: kick off / resume an optimization session -evals/ -└── sdlc-bench/ # NEW (Tier 3): fixture repos + task specs + held-out tests -``` - -1. **Tier 1** — `score` command + run ledger + genome extraction of the knobs that - already exist (budgets, review config, signs). No behavior change; pure telemetry. -2. **Tier 2** — loop reads `.sdlc/workflow.json`; `signs.md` becomes the inbox. -3. **Tier 3 (benchmark)** — `evals/sdlc-bench/` suite + autoloop-generated runner. -4. **Tier 3 (search)** — the `sdlc-optimize` skill; dogfood one overnight session on - the bench suite before touching shadow mode. +Weights are a starting point, to be sanity-checked against the human's own sense of +"that run went well" once a dozen records exist — before any retro acts on them. + +**Secondary metrics — tracked, never optimized** (the Goodhart guard, borrowed from +autoloop): wall-clock, `wait_ticks`, decision count, and (future) post-merge signal — +human review comments on the loop's PRs, reverts of loop commits. A skill change that +raises scores while secondary metrics degrade is a red flag, not a win. + +## 5. Retro: Periodically Feed the Results Back + +A new `/sdlc-retro` command + `sdlc-retro` skill. Human-invoked ("periodically" is a +human judgment); the SHIP step may *nudge* — "14 runs since the last retro, consider +`/sdlc-retro`" — but never self-invokes. + +One retro session: + +1. **Load** all ledger lines since the last retro marker; drill into the archive + directories of the worst-scoring runs (the failure log, in AFlow's sense). +2. **Mine for patterns** across runs — the checklist, roughly in order of value: + - Recurring **BLOCKED reasons** (budget? no-progress? same escalation shape twice?) + - **Rework clusters** — which state pairs bounce most; what the transition `reason` + strings have in common (e.g. "REVIEW keeps finding missing error handling" → + candidate Builder-prompt or sign change). + - **Attempts-exceeded** task shapes — what kinds of tasks burn 3 strikes. + - **Iteration sinks** — states consuming disproportionate ticks vs. work produced + (e.g. chronic `wait_ticks` → the waiting guidance isn't landing). + - **Sign audit** — for each active sign, compare scores of runs with vs. without + it. Signs are finally *removable*: an unhelpful one becomes a removal proposal. + - **Decision audit** — decisions that later correlate with REPAIR/rework. +3. **Propose ≤5 concrete changes**, each citing the runs that motivate it + (`runs.jsonl` line refs). Eligible targets — the plugin source in this repo: + - `sdlc-loop/SKILL.md` dispatch-table prose (the workflow itself), + - default budgets and review-gate defaults in `sdlc_state.py`, + - Architect/Builder agent prompts, + - sign graduation into skill text, or sign removal, + - the score weights themselves (flagged as such — changing the ruler is allowed + but must be its own proposal, never bundled with changes measured by it). +4. **Deliver as a PR** against `claude-plugins` on a branch, one commit per proposal, + with the evidence in the PR body. The human reviews and merges — the retro **never + self-merges** (§7). A `retro.json` marker (date, ledger position, plugin version, + proposals) is written so the next retro knows its window and what to measure. + +**Consumer mode.** Marketplace users who can't edit the plugin get the same retro with +a different output channel: proposals land as feedback-skill entries and `signs.md` +candidates (the existing `feedback save`/`consolidate` path) instead of a source PR. +Author mode is the primary target — it is how the skill itself compounds. + +## 6. Measure: Did the Last Change Help? + +This is AFlow's "keep only if improved," amortized over real usage. Each retro's first +act is to grade the previous one: + +- Partition ledger lines by `plugin_version` (before vs. after the last merged retro + PR); compare score windows — pessimistically (`mean − stderr`), with a minimum n + (default 5 runs) before any conclusion is drawn. +- Improved or neutral → note it in the new retro PR and move on. +- Regressed → the first proposal of the new retro is a **revert** of the offending + change, with the two windows as evidence. + +Honest caveats, and why this is still enough: real features are not i.i.d. — a bad +fortnight of gnarly tasks can sink a window. The pessimistic comparison, the minimum +n, and the fact that a human reviews every conclusion (this is a PR gate, not an +autonomous actuator) keep noise from compounding. What this design gives up relative +to benchmark-driven search is *statistical crispness*; what it gains is zero marginal +evaluation cost and scores grounded in the work the user actually cares about. + +## 7. Guardrails + +- **The retro proposes; the human merges.** The optimizer path never edits skill + text, agents, or hooks directly — a system that auto-approves its own actions must + not also choose its own rules. The PR is the boundary. +- **One proposal, one commit, one measurable claim** — so a regression found later is + revertible at proposal granularity. +- **Secondary metrics veto, never climb** (§4). +- **No mid-run mutation**: whatever skill version a run starts with, it keeps; scores + attribute cleanly via `plugin_version`. +- **Patterns need ≥2 supporting runs.** A single pathological run makes a `signs.md` + candidate at most, never a skill-text proposal — that is the existing signs channel + doing its job at the right altitude. + +## 8. Future Work: Deliberate Search (the original AFlow shape) + +If passive accumulation plateaus — retros keep coming back empty while scores sit +below where they should be — the ledger and score function built here are exactly the +substrate real search needs. The earlier draft of this note specified it: reify the +tunable surface as a hashable *workflow genome* (`.sdlc/workflow.json`: budgets, +review config, signs, bounded per-state prompt overlays); build a small +`evals/sdlc-bench/` fixture suite whose **held-out acceptance tests** restore ground +truth; drive MCTS-lite over genomes (AFlow's soft-mixture selection over pessimistic +top-k, schema-constrained expansion, keep-if-improved, top-k-plateau stopping) with +**autoloop** as the evaluator harness — the genome is the mutable artifact, the bench +runner emits `METRIC score=…`. Roughly 10 rounds × 3 tasks ≈ 30 headless loop runs, +an overnight-shaped session. None of it is needed until the cheap loop stops paying. + +## 9. Migration + +1. **Capture + score** — `sdlc_state.py`: archive-on-terminal-transition, `score` + command, ledger append. Pure telemetry; no behavior change; test alongside the + existing `test_sdlc_state.py` suite. +2. **Ledger bake-in** — run real features for a while; sanity-check scores against + human judgment; tune weights once. +3. **`/sdlc-retro`** — command + skill (author mode first), including the + measure-previous-retro step and the `retro.json` marker. +4. **Consumer mode** — feedback-skill output channel. +5. **(Optional, later)** genome + bench suite + MCTS per §8. ## Sources - [AFlow: Automating Agentic Workflow Generation (Zhang et al., ICLR 2025)](https://arxiv.org/abs/2410.10762) -- `docs/sdlc-loop-redesign.md` — the loop being optimized, and its signs/feedback mechanism -- `plugins/autoloop/skills/autoloop/SKILL.md` — the seven-component evaluator harness reused by benchmark mode +- `docs/sdlc-loop-redesign.md` — the loop, its signs/feedback mechanism, and the state CLI this extends +- `docs/compound-engineering-roadmap.md` — "every unit of work should make the next unit easier"; this note is that principle applied to the loop itself +- `plugins/autoloop/skills/autoloop/SKILL.md` — secondary-metric guardrails; the evaluator harness for §8 - [Anthropic Engineering — Effective harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) From 22f713b994ae7fe653bd3fc81d3be413e07e084e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 9 Jul 2026 15:55:13 +0000 Subject: [PATCH 3/3] feat(autonomous-sdlc): run capture, scoring, and /sdlc-retro (AFlow PoC) Proof of concept for docs/aflow-sdlc-optimization.md: - sdlc_state.py: every terminal transition (DONE and BLOCKED, including budget force-blocks) archives the run's full trace to the user-level ledger at ~/.claude/autonomous-sdlc/ (override: SDLC_RUNS_DIR) and appends a scored runs.jsonl line carrying plugin_version. New 'score' subcommand computes the composite (outcome, efficiency, rework rate, autonomy) from the current increment's transition history. Archiving is best-effort and can never break a transition. - sdlc_retro.py: 'digest' aggregates the ledger since the last retro marker (per-version score windows with pessimistic means and a min-n comparability gate, outcome/rework totals, worst-run pointers); 'mark' records a retro point. - New sdlc-retro skill + /sdlc-retro command: grade the previous retro, mine worst runs, deliver <=5 grounded proposals (>=2 supporting runs each) as a PR in author mode or feedback entries in consumer mode. - sdlc-loop skill: notes automatic archiving at SHIP->DONE and nudges /sdlc-retro at 10+ unretroed runs. - Tests for scoring, archive-on-terminal, archive-failure safety, increment windowing, and the digest/mark CLI. Plugin 2.3.1 -> 2.4.0. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01UT9Qjmu44qeUVEbwfbSfMk --- .claude-plugin/marketplace.json | 2 +- CLAUDE.md | 2 +- docs/aflow-sdlc-optimization.md | 5 +- .../.claude-plugin/plugin.json | 2 +- plugins/autonomous-sdlc/README.md | 28 ++- .../autonomous-sdlc/commands/sdlc-retro.md | 22 +++ plugins/autonomous-sdlc/scripts/sdlc_retro.py | 183 ++++++++++++++++++ plugins/autonomous-sdlc/scripts/sdlc_state.py | 150 ++++++++++++++ .../scripts/test_sdlc_retro.py | 150 ++++++++++++++ .../scripts/test_sdlc_state.py | 140 +++++++++++++- .../autonomous-sdlc/skills/sdlc-loop/SKILL.md | 10 +- .../skills/sdlc-retro/SKILL.md | 117 +++++++++++ 12 files changed, 801 insertions(+), 10 deletions(-) create mode 100644 plugins/autonomous-sdlc/commands/sdlc-retro.md create mode 100644 plugins/autonomous-sdlc/scripts/sdlc_retro.py create mode 100644 plugins/autonomous-sdlc/scripts/test_sdlc_retro.py create mode 100644 plugins/autonomous-sdlc/skills/sdlc-retro/SKILL.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index dbbd1c6..d6ffb81 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -30,7 +30,7 @@ "name": "autonomous-sdlc", "source": "./plugins/autonomous-sdlc", "description": "Autonomous SDLC as a state machine on disk driven by a Stop-hook loop (optionally a user-armed /goal) \u2014 one verified unit of work per iteration, decide-log-proceed autonomy, built-in verify/code-review/simplify gates", - "version": "2.3.1", + "version": "2.4.0", "author": { "name": "Joshua Oliphant" }, diff --git a/CLAUDE.md b/CLAUDE.md index 09efb55..9677148 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,7 @@ Each plugin owns its version in `plugins/{name}/.claude-plugin/plugin.json` (the | Plugin | Purpose | Notable pieces | |---|---|---| | **mochi-creator** | Create cognitive-science-based flashcards via the Mochi API | `scripts/mochi_api.py` (module + CLI), prompt-quality validation, knowledge-type references | -| **autonomous-sdlc** | Autonomous SDLC as a state machine on disk driven by a loop | `sdlc-loop` skill + `sdlc_state.py` state CLI, 6 skills, 2 subagents (Architect/Builder), loop Stop hook + denylisted auto-approve | +| **autonomous-sdlc** | Autonomous SDLC as a state machine on disk driven by a loop | `sdlc-loop` skill + `sdlc_state.py` state CLI, 7 skills, 2 subagents (Architect/Builder), loop Stop hook + denylisted auto-approve, scored run ledger + `/sdlc-retro` | | **hexagonal-agents** | Web apps where an agent generates HTML UI | Ports-and-adapters arch, MCP tools, Claude Agent SDK, extensive `references/` | | **compound-knowledge** | Institutional memory: capture → retrieve → graduate | YAML-frontmatter solution files, grep-based retrieval, `knowledge-researcher` subagent | | **autoloop** | Generate Karpathy-style optimization loops | Produces `program.md` + immutable `auto/run.sh`, `codebase-scout` subagent | diff --git a/docs/aflow-sdlc-optimization.md b/docs/aflow-sdlc-optimization.md index 000c5cc..09f6d3e 100644 --- a/docs/aflow-sdlc-optimization.md +++ b/docs/aflow-sdlc-optimization.md @@ -3,7 +3,10 @@ # Learning From Every Run: Feeding SDLC Traces Back Into the Skill -**Status**: Proposed design (not yet implemented) +**Status**: Proof of concept implemented in autonomous-sdlc v2.4.0 — capture + score +(`sdlc_state.py score` / archive-on-terminal-transition), retro tooling +(`sdlc_retro.py digest|mark`), and the `/sdlc-retro` skill. The measure loop (§6) +activates once real ledger data accumulates. **Target**: autonomous-sdlc (post-v2.1) **Companion**: `docs/sdlc-loop-redesign.md` (the loop being improved), `docs/compound-engineering-roadmap.md` (this is the "compound" step applied to the loop itself) diff --git a/plugins/autonomous-sdlc/.claude-plugin/plugin.json b/plugins/autonomous-sdlc/.claude-plugin/plugin.json index 33fa9e4..b371f1f 100644 --- a/plugins/autonomous-sdlc/.claude-plugin/plugin.json +++ b/plugins/autonomous-sdlc/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "autonomous-sdlc", - "version": "2.3.1", + "version": "2.4.0", "description": "Autonomous SDLC as a state machine on disk driven by a Stop-hook loop (optionally a user-armed /goal) \u2014 one verified unit of work per iteration, decide-log-proceed autonomy, built-in verify/code-review/simplify gates", "author": { "name": "Joshua Oliphant", diff --git a/plugins/autonomous-sdlc/README.md b/plugins/autonomous-sdlc/README.md index c79da9e..c7fa369 100644 --- a/plugins/autonomous-sdlc/README.md +++ b/plugins/autonomous-sdlc/README.md @@ -102,6 +102,7 @@ worktree isolation. PR creation is one `gh pr create` call. | `/autonomous-sdlc:sdlc-status` | Render `.sdlc/state.json` + progress | | `/autonomous-sdlc:sdlc-cancel` | Transition to BLOCKED(cancelled); state kept for resume | | `/autonomous-sdlc:prime` | Orient to a codebase (outside the loop) | +| `/autonomous-sdlc:sdlc-retro` | Retro over the run ledger — propose skill improvements as a reviewable PR | ## Skills @@ -113,6 +114,7 @@ worktree isolation. PR creation is one `gh pr create` call. | `tdd-workflow` | Red-green-refactor inner loop | | `beads-workflow` | Beads task graph — the loop's work queue (`bd ready` is the wave) | | `feedback` | Cross-session preferences + graduation target for loop signs | +| `sdlc-retro` | Mine archived run traces for cross-run patterns; deliver grounded, human-reviewed improvement proposals | ## State Files (in the target project) @@ -129,7 +131,13 @@ specs/{slug}-plan.md # architect plan `python3 scripts/sdlc_state.py --help` documents the state CLI (init, tick [--waiting], transition, increment, task [--done], attempt, decide, note-progress, set-budget, -set-driver, status, state). +set-driver, status, state, score). + +Every run that reaches DONE or BLOCKED is automatically archived — trace files plus a +scored ledger line — to the user-level `~/.claude/autonomous-sdlc/` (override: +`SDLC_RUNS_DIR`), so the loop learns across projects instead of discarding its history. +`scripts/sdlc_retro.py digest` aggregates the ledger for `/sdlc-retro`; design: +[`docs/aflow-sdlc-optimization.md`](../../docs/aflow-sdlc-optimization.md). ## Composes With (soft dependencies — skipped silently when absent) @@ -188,7 +196,23 @@ BUILD. A blank `--reviewers` value falls back to the default so the gate is neve ## Version History -### v2.3.0 (Current) +### v2.4.0 (Current) +- **Run capture + scoring (the retro ledger)**: every terminal run (DONE *and* BLOCKED, + including budget force-blocks) archives its full trace (`state.json` history, + `decisions.jsonl`, `progress.md`, `signs.md`, `escalation.md`) to + `~/.claude/autonomous-sdlc/runs/` with a composite-scored line in `runs.jsonl` + (outcome, efficiency, rework rate, autonomy; `plugin_version` attributes scores to the + skill version that produced them). New `score` subcommand; archiving is best-effort and + can never break a transition. +- **`/sdlc-retro` + `sdlc-retro` skill**: periodic, human-invoked retrospective — grade + the previous retro by comparing per-version score windows (pessimistic, min-n gated), + mine the worst runs for cross-run patterns, and deliver ≤5 grounded proposals (own + commit each) as a PR in author mode or feedback entries in consumer mode. Backed by + `scripts/sdlc_retro.py` (`digest`/`mark`). The retro proposes; the human merges. +- Design note: `docs/aflow-sdlc-optimization.md` (AFlow's score/ledger/expand/keep loop, + amortized over real runs). + +### v2.3.0 - **Next-increment lifecycle** (the loop is no longer single-use per project): a finished (`DONE`) session re-invoked with a new feature now starts increment 2 instead of silently resuming `DONE` and dropping the request. New `increment` subcommand archives the finished diff --git a/plugins/autonomous-sdlc/commands/sdlc-retro.md b/plugins/autonomous-sdlc/commands/sdlc-retro.md new file mode 100644 index 0000000..4a6b252 --- /dev/null +++ b/plugins/autonomous-sdlc/commands/sdlc-retro.md @@ -0,0 +1,22 @@ +--- +name: sdlc-retro +description: Retrospective over archived SDLC runs — mine the run ledger and propose skill improvements as a reviewable PR +allowed-tools: + - Bash + - Read + - Glob + - Grep + - Write + - Edit + - Skill +--- + +# SDLC Retro + +Run the `sdlc-retro` skill: digest the run ledger at `~/.claude/autonomous-sdlc/`, +grade the previous retro, mine the archived traces for cross-run patterns, and +deliver ≤5 grounded improvement proposals (PR in author mode, feedback entries in +consumer mode), then mark the retro. + +Do not run this inside an active loop iteration — it is the between-features +feedback step, invoked by the human. diff --git a/plugins/autonomous-sdlc/scripts/sdlc_retro.py b/plugins/autonomous-sdlc/scripts/sdlc_retro.py new file mode 100644 index 0000000..7c1684e --- /dev/null +++ b/plugins/autonomous-sdlc/scripts/sdlc_retro.py @@ -0,0 +1,183 @@ +#!/usr/bin/env python3 +# ABOUTME: Retro CLI over the autonomous-sdlc run ledger (~/.claude/autonomous-sdlc). +# ABOUTME: `digest` aggregates runs since the last retro marker; `mark` records a retro point. +"""Aggregate the run ledger for /sdlc-retro. + +The loop archives every terminal run (sdlc_state.py archive_run) into a +user-level ledger. This CLI turns that ledger into the digest the sdlc-retro +skill reasons over: score windows per plugin version, outcome and rework +aggregates, and pointers to the worst runs' archived traces. It computes; the +skill (and the human reviewing its PR) interpret. + +Usage: + python3 sdlc_retro.py digest [--all] [--worst N] + python3 sdlc_retro.py mark --note "" +""" + +import argparse +import json +import math +import os +import sys +from collections import Counter, defaultdict +from datetime import datetime, timezone +from pathlib import Path + +# Minimum runs on BOTH sides before a version-window comparison means anything +# (docs/aflow-sdlc-optimization.md §6). +MIN_WINDOW_N = 5 + + +def runs_root() -> Path: + env = os.environ.get("SDLC_RUNS_DIR") + return Path(env) if env else Path.home() / ".claude" / "autonomous-sdlc" + + +def now() -> str: + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +def load_ledger(root: Path) -> list[dict]: + ledger_file = root / "runs.jsonl" + if not ledger_file.exists(): + return [] + records = [] + for line in ledger_file.read_text().splitlines(): + if not line.strip(): + continue + try: + records.append(json.loads(line)) + except ValueError: + print(f"WARN skipping corrupt ledger line: {line[:80]}", file=sys.stderr) + return records + + +def load_marker(root: Path) -> dict | None: + marker_file = root / "retro.json" + if not marker_file.exists(): + return None + try: + return json.loads(marker_file.read_text()) + except ValueError: + print("WARN retro.json is corrupt; treating as no marker", file=sys.stderr) + return None + + +def window_stats(scores: list[float]) -> dict: + n = len(scores) + mean = sum(scores) / n + var = sum((x - mean) ** 2 for x in scores) / n + stderr = math.sqrt(var / n) if n > 1 else 0.0 + return { + "n": n, + "mean": round(mean, 3), + # Pessimistic score: what comparisons should use, so an n=1 fluke + # never displaces an established window. + "pessimistic": round(mean - stderr, 3), + "min": round(min(scores), 3), + "max": round(max(scores), 3), + "comparable": n >= MIN_WINDOW_N, + } + + +def cmd_digest(args: argparse.Namespace) -> None: + root = runs_root() + ledger = load_ledger(root) + marker = load_marker(root) + since = 0 if (args.all or marker is None) else marker.get("ledger_lines", 0) + window = ledger[since:] + + digest: dict = { + "ledger_total": len(ledger), + "runs": len(window), + "since_last_retro": not args.all and marker is not None, + "previous_retro": marker, + } + if not window: + print(json.dumps(digest, indent=2)) + return + + scores = [r["score"] for r in window] + outcomes = Counter(r["outcome"] for r in window) + blocked_reasons = Counter( + r.get("terminal_reason", "?") for r in window if r["outcome"] == "BLOCKED" + ) + rework_totals: Counter = Counter() + for r in window: + rework_totals.update(r.get("rework", {})) + by_version: dict[str, list[float]] = defaultdict(list) + for r in window: + by_version[r.get("plugin_version", "unknown")].append(r["score"]) + + digest.update( + { + "score": window_stats(scores), + "outcomes": dict(outcomes), + "blocked_reasons": dict(blocked_reasons.most_common()), + "rework_totals": dict(rework_totals), + "attempts_exceeded_total": sum( + r.get("attempts_exceeded", 0) for r in window + ), + # Whole-ledger view so a retro can grade its predecessor even when + # the pre-change runs fall outside the current window. + "by_plugin_version": { + v: window_stats(s) + for v, s in sorted( + ((v, [r["score"] for r in ledger if r.get("plugin_version") == v]) + for v in {r.get("plugin_version", "unknown") for r in ledger}), + ) + }, + "worst_runs": [ + { + "at": r["at"], + "repo": r.get("repo"), + "feature": r.get("feature"), + "score": r["score"], + "outcome": r["outcome"], + "terminal_reason": r.get("terminal_reason"), + "archive": r.get("archive"), + } + for r in sorted(window, key=lambda r: r["score"])[: args.worst] + ], + } + ) + print(json.dumps(digest, indent=2)) + + +def cmd_mark(args: argparse.Namespace) -> None: + root = runs_root() + root.mkdir(parents=True, exist_ok=True) + ledger = load_ledger(root) + marker = { + "at": now(), + "ledger_lines": len(ledger), + "plugin_version": ( + ledger[-1].get("plugin_version", "unknown") if ledger else "unknown" + ), + "note": args.note, + } + (root / "retro.json").write_text(json.dumps(marker, indent=2) + "\n") + with (root / "retros.jsonl").open("a") as f: + f.write(json.dumps(marker) + "\n") + print(f"OK retro marked at ledger line {marker['ledger_lines']}") + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__) + sub = p.add_subparsers(dest="cmd", required=True) + + sp = sub.add_parser("digest", help="aggregate runs since the last retro marker") + sp.add_argument("--all", action="store_true", help="ignore the retro marker") + sp.add_argument("--worst", type=int, default=3, help="worst runs to surface") + sp.set_defaults(func=cmd_digest) + + sp = sub.add_parser("mark", help="record that a retro ran (windows the next one)") + sp.add_argument("--note", required=True, help="one-line summary of the retro") + sp.set_defaults(func=cmd_mark) + + args = p.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/plugins/autonomous-sdlc/scripts/sdlc_state.py b/plugins/autonomous-sdlc/scripts/sdlc_state.py index 8813c84..754996e 100644 --- a/plugins/autonomous-sdlc/scripts/sdlc_state.py +++ b/plugins/autonomous-sdlc/scripts/sdlc_state.py @@ -29,6 +29,8 @@ import argparse import json +import os +import shutil import sys from datetime import datetime, timezone from pathlib import Path @@ -37,6 +39,8 @@ STATE_FILE = SDLC_DIR / "state.json" DECISIONS_FILE = SDLC_DIR / "decisions.jsonl" PROGRESS_FILE = SDLC_DIR / "progress.md" +SIGNS_FILE = SDLC_DIR / "signs.md" +ESCALATION_FILE = SDLC_DIR / "escalation.md" STATES = [ "INIT", @@ -106,6 +110,141 @@ def now() -> str: return datetime.now(timezone.utc).isoformat(timespec="seconds") +# --- Run capture + scoring (docs/aflow-sdlc-optimization.md §§3-4) ------------ +# +# Every run that reaches a terminal state (DONE or BLOCKED, including budget +# force-blocks) archives its full trace to a durable user-level ledger instead +# of throwing it away. The ledger is what /sdlc-retro periodically mines to +# propose improvements to the skill itself. + +# v1 heuristic: a task typically costs ~3 iterations (build + its share of +# verify/review). Tune only after sanity-checking a dozen real ledger records +# against human judgment of "that run went well". +BASELINE_ITERATIONS_PER_TASK = 3.0 + + +def runs_root() -> Path: + """User-level, cross-project archive. SDLC_RUNS_DIR overrides (tests).""" + env = os.environ.get("SDLC_RUNS_DIR") + return Path(env) if env else Path.home() / ".claude" / "autonomous-sdlc" + + +def plugin_version() -> str: + """The installed plugin's version — attributes ledger scores to the exact + skill version that produced them, so a retro can measure its predecessor.""" + manifest = Path(__file__).resolve().parent.parent / ".claude-plugin" / "plugin.json" + try: + return json.loads(manifest.read_text())["version"] + except (OSError, ValueError, KeyError): + return "unknown" + + +def current_increment_history(state: dict) -> list[dict]: + """History entries for the current increment: from its INIT entry on. + Earlier increments were archived at their own terminal transition.""" + hist = state["history"] + start = 0 + for i, h in enumerate(hist): + if h["to"] == "INIT": + start = i + return hist[start:] + + +def compute_score(state: dict) -> dict: + """Composite run score from the transition trace. Weights per the design + note: outcome dominates; efficiency, rework, and autonomy split the rest.""" + states = [h["to"] for h in current_increment_history(state)] + pairs = list(zip(states, states[1:])) + rework = { + "verify_bounces": pairs.count(("VERIFY", "BUILD")), + "review_roundtrips": pairs.count(("REVIEW", "BUILD")), + "replans": pairs.count(("PLAN", "PLAN")), + "repairs": states.count("REPAIR"), + } + rework_rate = min(sum(rework.values()) / max(len(pairs), 1), 1.0) + + attempts = state.get("attempts", {}) + tasks = max(len(attempts), 1) + limit = state["budgets"]["max_attempts_per_task"] + attempts_exceeded = sum(1 for n in attempts.values() if n > limit) + + iterations = state.get("iteration", 0) + per_task = iterations / tasks + efficiency = ( + 1.0 + if per_task <= BASELINE_ITERATIONS_PER_TASK + else BASELINE_ITERATIONS_PER_TASK / per_task + ) + outcome = 1.0 if state["state"] == "DONE" else 0.0 + autonomy = 1.0 - min(attempts_exceeded / tasks, 1.0) + score = ( + 0.5 * outcome + + 0.2 * efficiency + + 0.2 * (1.0 - rework_rate) + + 0.1 * autonomy + ) + return { + "score": round(score, 3), + "outcome": state["state"], + "iterations": iterations, + "tasks_attempted": len(attempts), + "attempts_exceeded": attempts_exceeded, + "rework": rework, + "rework_rate": round(rework_rate, 3), + "wait_ticks": state.get("wait_ticks", 0), + } + + +def archive_run(state: dict, reason: str) -> None: + """Copy the run's trace files into the archive and append a ledger line. + + Best-effort by design: the terminal transition must succeed even when + archiving cannot (read-only home, weird cwd), so failures only warn. + """ + try: + root = runs_root() + stamp = now().replace(":", "-").replace("+00-00", "Z") + slug = f"{stamp}_{Path.cwd().name}_{state['feature']}".replace("/", "-") + run_dir = root / "runs" / slug + suffix = 1 + while run_dir.exists(): + suffix += 1 + run_dir = root / "runs" / f"{slug}-{suffix}" + run_dir.mkdir(parents=True) + for f in (STATE_FILE, DECISIONS_FILE, PROGRESS_FILE, SIGNS_FILE, ESCALATION_FILE): + if f.exists(): + shutil.copy2(f, run_dir / f.name) + signs_active = 0 + if SIGNS_FILE.exists(): + signs_active = sum( + 1 + for line in SIGNS_FILE.read_text().splitlines() + if line.strip() and not line.lstrip().startswith("#") + ) + decisions = ( + len(DECISIONS_FILE.read_text().splitlines()) + if DECISIONS_FILE.exists() + else 0 + ) + record = { + "at": now(), + "repo": Path.cwd().name, + "feature": state["feature"], + "cycle": state.get("cycle", 1), + "plugin_version": plugin_version(), + "terminal_reason": reason, + "decisions": decisions, + "signs_active": signs_active, + "archive": str(run_dir), + **compute_score(state), + } + with (root / "runs.jsonl").open("a") as f: + f.write(json.dumps(record) + "\n") + print(f"ARCHIVED {run_dir}") + except Exception as e: # noqa: BLE001 — never let archiving break the loop + print(f"WARN archive failed: {e}", file=sys.stderr) + + def load() -> dict: if not STATE_FILE.exists(): sys.exit("No .sdlc/state.json — run `sdlc_state.py init` first.") @@ -222,6 +361,7 @@ def block(state: dict, reason: str) -> None: state["state"] = "BLOCKED" save(state) append_progress(f"BLOCKED: {reason}") + archive_run(state, reason) print(f"BLOCKED {reason}") @@ -323,6 +463,8 @@ def cmd_transition(args: argparse.Namespace) -> None: s["last_progress_iteration"] = s["iteration"] save(s) append_progress(f"→ {target}: {args.reason}") + if target in ("DONE", "BLOCKED"): + archive_run(s, args.reason) print(f"OK {target}") @@ -396,6 +538,10 @@ def cmd_set_driver(args: argparse.Namespace) -> None: print(f"OK driver={args.driver}") +def cmd_score(_args: argparse.Namespace) -> None: + print(json.dumps(compute_score(load()), indent=2)) + + def cmd_note_progress(args: argparse.Namespace) -> None: s = load() s["last_progress_iteration"] = s["iteration"] @@ -490,6 +636,10 @@ def main() -> None: sp.add_argument("--what", required=True) sp.set_defaults(func=cmd_note_progress) + sub.add_parser( + "score", help="composite run score computed from the transition trace" + ).set_defaults(func=cmd_score) + args = p.parse_args() args.func(args) diff --git a/plugins/autonomous-sdlc/scripts/test_sdlc_retro.py b/plugins/autonomous-sdlc/scripts/test_sdlc_retro.py new file mode 100644 index 0000000..a178537 --- /dev/null +++ b/plugins/autonomous-sdlc/scripts/test_sdlc_retro.py @@ -0,0 +1,150 @@ +# ABOUTME: Tests for sdlc_retro.py — the run-ledger digest and retro marker CLI. +# ABOUTME: Covers windowing by marker, per-version stats, and worst-run surfacing. +"""Run from this directory: `python3 -m pytest test_sdlc_retro.py` (or plain +`python3 test_sdlc_retro.py` for the bundled fallback runner).""" + +import importlib.util +import io +import json +import os +from contextlib import redirect_stdout +from pathlib import Path + +_SPEC = importlib.util.spec_from_file_location( + "sdlc_retro", Path(__file__).with_name("sdlc_retro.py") +) +sdlc_retro = importlib.util.module_from_spec(_SPEC) +_SPEC.loader.exec_module(sdlc_retro) + + +def _record(score, outcome="DONE", version="2.4.0", reason="pr", **extra): + return { + "at": "2026-07-08T00:00:00+00:00", + "repo": "demo", + "feature": "f", + "plugin_version": version, + "terminal_reason": reason, + "outcome": outcome, + "score": score, + "rework": {"verify_bounces": 1, "review_roundtrips": 0, + "replans": 0, "repairs": 0}, + "attempts_exceeded": 0, + "archive": "/tmp/x", + **extra, + } + + +def _write_ledger(root: Path, records): + root.mkdir(parents=True, exist_ok=True) + with (root / "runs.jsonl").open("w") as f: + for r in records: + f.write(json.dumps(r) + "\n") + + +def _digest(root, **ns): + os.environ["SDLC_RUNS_DIR"] = str(root) + try: + out = io.StringIO() + args = sdlc_retro.argparse.Namespace( + all=ns.get("all", False), worst=ns.get("worst", 3) + ) + with redirect_stdout(out): + sdlc_retro.cmd_digest(args) + return json.loads(out.getvalue()) + finally: + del os.environ["SDLC_RUNS_DIR"] + + +def _mark(root, note="n"): + os.environ["SDLC_RUNS_DIR"] = str(root) + try: + out = io.StringIO() + with redirect_stdout(out): + sdlc_retro.cmd_mark(sdlc_retro.argparse.Namespace(note=note)) + finally: + del os.environ["SDLC_RUNS_DIR"] + + +def test_digest_empty_ledger(tmp_path): + digest = _digest(tmp_path) + assert digest["runs"] == 0 + assert digest["previous_retro"] is None + + +def test_digest_aggregates_and_surfaces_worst(tmp_path): + _write_ledger( + tmp_path, + [ + _record(0.9), + _record(0.2, outcome="BLOCKED", reason="budget: max_iterations=50"), + _record(0.7), + ], + ) + digest = _digest(tmp_path, worst=1) + assert digest["runs"] == 3 + assert digest["outcomes"] == {"DONE": 2, "BLOCKED": 1} + assert digest["blocked_reasons"] == {"budget: max_iterations=50": 1} + assert digest["rework_totals"]["verify_bounces"] == 3 + assert len(digest["worst_runs"]) == 1 + assert digest["worst_runs"][0]["score"] == 0.2 + assert digest["score"]["n"] == 3 + assert digest["score"]["comparable"] is False # below MIN_WINDOW_N + + +def test_mark_windows_the_next_digest(tmp_path): + _write_ledger(tmp_path, [_record(0.5), _record(0.6)]) + _mark(tmp_path, note="first retro") + # Two more runs arrive after the retro. + with (tmp_path / "runs.jsonl").open("a") as f: + f.write(json.dumps(_record(0.8)) + "\n") + f.write(json.dumps(_record(0.9)) + "\n") + + digest = _digest(tmp_path) + assert digest["since_last_retro"] is True + assert digest["runs"] == 2 # only post-marker runs + assert digest["ledger_total"] == 4 + assert digest["previous_retro"]["note"] == "first retro" + # --all ignores the marker. + assert _digest(tmp_path, all=True)["runs"] == 4 + + +def test_version_windows_span_whole_ledger(tmp_path): + # Grading a previous retro needs pre-change runs even when the marker + # window excludes them: by_plugin_version must cover the whole ledger. + records = [_record(0.4, version="2.3.1") for _ in range(5)] + records += [_record(0.8, version="2.4.0") for _ in range(5)] + _write_ledger(tmp_path, records) + _mark(tmp_path) + with (tmp_path / "runs.jsonl").open("a") as f: + f.write(json.dumps(_record(0.9, version="2.4.0")) + "\n") + + digest = _digest(tmp_path) + versions = digest["by_plugin_version"] + assert versions["2.3.1"]["n"] == 5 + assert versions["2.4.0"]["n"] == 6 + assert versions["2.3.1"]["comparable"] and versions["2.4.0"]["comparable"] + assert versions["2.4.0"]["pessimistic"] > versions["2.3.1"]["pessimistic"] + + +if __name__ == "__main__": + # Minimal fallback runner so the suite works without pytest installed. + import tempfile + import traceback + + tests = [ + test_digest_empty_ledger, + test_digest_aggregates_and_surfaces_worst, + test_mark_windows_the_next_digest, + test_version_windows_span_whole_ledger, + ] + failures = 0 + for t in tests: + with tempfile.TemporaryDirectory() as d: + try: + t(Path(d)) + print(f"PASS {t.__name__}") + except Exception: + failures += 1 + print(f"FAIL {t.__name__}") + traceback.print_exc() + raise SystemExit(1 if failures else 0) diff --git a/plugins/autonomous-sdlc/scripts/test_sdlc_state.py b/plugins/autonomous-sdlc/scripts/test_sdlc_state.py index 4b7a663..fd39ecf 100644 --- a/plugins/autonomous-sdlc/scripts/test_sdlc_state.py +++ b/plugins/autonomous-sdlc/scripts/test_sdlc_state.py @@ -285,6 +285,140 @@ def test_resume_preserves_existing_review_block(tmp_path): assert after == before +# --- Run capture + scoring (the /sdlc-retro ledger) ------------------------- + + +def _terminal_via_transition(tmp_path, target, reason): + """Drive the state to `target` through cmd_transition (legal edge required).""" + _run(tmp_path, sdlc_state.cmd_transition, target=target, reason=reason) + + +def test_score_reflects_outcome_and_rework(tmp_path): + _init(tmp_path) + s = _read_state(tmp_path) + s["state"] = "DONE" + s["iteration"] = 12 + s["attempts"] = {"bd-1": 1, "bd-2": 2, "bd-3": 4, "bd-4": 1} # bd-3 exceeded + s["history"] = [ + {"at": "t", "to": st, "reason": "r"} + for st in [ + "INIT", "SPEC", "PLAN", "BUILD", "VERIFY", "BUILD", # verify bounce + "VERIFY", "REVIEW", "BUILD", "VERIFY", "REVIEW", # review roundtrip + "SHIP", "DONE", + ] + ] + _write_state(tmp_path, s) + metrics = sdlc_state.compute_score(_read_state(tmp_path)) + assert metrics["outcome"] == "DONE" + assert metrics["rework"] == { + "verify_bounces": 1, + "review_roundtrips": 1, + "replans": 0, + "repairs": 0, + } + assert metrics["tasks_attempted"] == 4 + assert metrics["attempts_exceeded"] == 1 + # 12 iterations / 4 tasks = 3.0/task → full efficiency; 2/12 backward edges. + assert 0.0 < metrics["score"] <= 1.0 + blocked = dict(s, state="BLOCKED") + assert sdlc_state.compute_score(blocked)["score"] < metrics["score"], ( + "outcome must dominate: same trace but BLOCKED scores lower" + ) + + +def test_terminal_transition_archives_run(tmp_path): + archive_root = tmp_path / "archive-root" + os.environ["SDLC_RUNS_DIR"] = str(archive_root) + try: + _init(tmp_path, _feature="cart") + s = _read_state(tmp_path) + s["state"] = "SHIP" + _write_state(tmp_path, s) + (tmp_path / ".sdlc" / "signs.md").write_text("# Signs\n- Sign: check first\n") + _terminal_via_transition(tmp_path, "DONE", "https://example.com/pr/1") + + ledger = (archive_root / "runs.jsonl").read_text().splitlines() + assert len(ledger) == 1 + record = json.loads(ledger[0]) + assert record["feature"] == "cart" + assert record["outcome"] == "DONE" + assert record["terminal_reason"] == "https://example.com/pr/1" + assert record["signs_active"] == 1 # header line not counted + assert record["plugin_version"] not in ("", None) + run_dir = Path(record["archive"]) + assert (run_dir / "state.json").exists() + assert (run_dir / "signs.md").exists() + # The archived state is the terminal one, not a stale pre-save copy. + assert json.loads((run_dir / "state.json").read_text())["state"] == "DONE" + finally: + del os.environ["SDLC_RUNS_DIR"] + + +def test_forced_block_archives_run(tmp_path): + # Budget force-blocks go through block(), not cmd_transition — the most + # informative runs (failures) must be captured too. + archive_root = tmp_path / "archive-root" + os.environ["SDLC_RUNS_DIR"] = str(archive_root) + try: + _init(tmp_path) + s = _read_state(tmp_path) + s["state"] = "BUILD" + s["iteration"] = 50 # at the budget; next tick exceeds it + s["last_progress_iteration"] = 50 + _write_state(tmp_path, s) + import pytest + + with pytest.raises(SystemExit): + _run(tmp_path, sdlc_state.cmd_tick, waiting=False) + record = json.loads((archive_root / "runs.jsonl").read_text().splitlines()[0]) + assert record["outcome"] == "BLOCKED" + assert "max_iterations" in record["terminal_reason"] + assert record["score"] < 0.5 # BLOCKED forfeits the outcome half + finally: + del os.environ["SDLC_RUNS_DIR"] + + +def test_archive_failure_never_breaks_the_transition(tmp_path): + # Point the archive root at a *file* so mkdir fails: the transition must + # still land (archive is best-effort by design). + poison = tmp_path / "not-a-dir" + poison.write_text("occupied") + os.environ["SDLC_RUNS_DIR"] = str(poison) + try: + _init(tmp_path) + s = _read_state(tmp_path) + s["state"] = "SHIP" + _write_state(tmp_path, s) + _terminal_via_transition(tmp_path, "DONE", "pr") + assert _read_state(tmp_path)["state"] == "DONE" + finally: + del os.environ["SDLC_RUNS_DIR"] + + +def test_score_only_sees_current_increment(tmp_path): + # A messy increment 1 must not drag down increment 2's score: the trace + # window starts at the latest INIT entry. + _init(tmp_path) + s = _read_state(tmp_path) + s["history"] = [ + {"at": "t", "to": st, "reason": "r"} + for st in ["INIT", "SPEC", "PLAN", "BUILD", "VERIFY", "BUILD", "VERIFY", + "REVIEW", "BUILD", "VERIFY", "REVIEW", "SHIP", "DONE"] + ] + _write_state(tmp_path, s) + _run(tmp_path, sdlc_state.cmd_increment, feature="phase2", request="more") + s = _read_state(tmp_path) + s["state"] = "DONE" + _write_state(tmp_path, s) + metrics = sdlc_state.compute_score(_read_state(tmp_path)) + assert metrics["rework"] == { + "verify_bounces": 0, + "review_roundtrips": 0, + "replans": 0, + "repairs": 0, + } + + if __name__ == "__main__": # Minimal fallback runner so the suite works without pytest installed. import tempfile @@ -303,8 +437,12 @@ def test_resume_preserves_existing_review_block(tmp_path): test_init_on_done_with_new_feature_auto_increments, test_init_on_done_same_feature_just_resumes, test_init_on_in_progress_does_not_increment, + test_score_reflects_outcome_and_rework, + test_terminal_transition_archives_run, + test_archive_failure_never_breaks_the_transition, + test_score_only_sees_current_increment, ] - # The pytest.raises test needs pytest; skip it in fallback mode. + # The pytest.raises tests need pytest; skip them in fallback mode. failures = 0 for t in tests: with tempfile.TemporaryDirectory() as d: diff --git a/plugins/autonomous-sdlc/skills/sdlc-loop/SKILL.md b/plugins/autonomous-sdlc/skills/sdlc-loop/SKILL.md index e6c6b54..bd9fa4e 100644 --- a/plugins/autonomous-sdlc/skills/sdlc-loop/SKILL.md +++ b/plugins/autonomous-sdlc/skills/sdlc-loop/SKILL.md @@ -5,7 +5,7 @@ description: > what one unit of work the current state requires, how to verify it, and which transition to record. Trigger: an active `.sdlc/state.json` exists, or the /sdlc command invokes it. Not for ad-hoc use outside a loop. -version: 2.1.0 +version: 2.2.0 effort: high allowed-tools: - Bash @@ -180,8 +180,12 @@ Push the branch (`git push -u origin feature/{slug}`). Create the PR yourself (`gh pr create` / `glab mr create`) with: summary from the plan doc, AC checklist from the spec, and a **"Decisions made autonomously"** section rendered from `.sdlc/decisions.jsonl`. Append the PR URL to `.sdlc/progress.md`. `transition DONE ---reason ""`. Optionally suggest the built-in **loop** skill to the user for PR -babysitting (`/loop 10m check PR CI and address review comments`). Auth failures +--reason ""` — the terminal transition automatically archives the run's trace +and score to `~/.claude/autonomous-sdlc/` (the /sdlc-retro ledger; BLOCKED archives +too). Optionally suggest the built-in **loop** skill to the user for PR +babysitting (`/loop 10m check PR CI and address review comments`), and if +`python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sdlc_retro.py digest` shows 10+ runs since the +last retro, suggest `/sdlc-retro` in your final summary. Auth failures escalate — never store or guess credentials. ### REPAIR (→ BUILD | VERIFY) diff --git a/plugins/autonomous-sdlc/skills/sdlc-retro/SKILL.md b/plugins/autonomous-sdlc/skills/sdlc-retro/SKILL.md new file mode 100644 index 0000000..e4388c7 --- /dev/null +++ b/plugins/autonomous-sdlc/skills/sdlc-retro/SKILL.md @@ -0,0 +1,117 @@ +--- +name: sdlc-retro +description: > + Periodic retrospective over the autonomous-sdlc run ledger: mine archived run + traces for cross-run patterns and feed them back into the SDLC skill as + grounded, human-reviewed improvement proposals. Trigger: the /sdlc-retro + command, or the user asking to "review SDLC runs", "improve the SDLC loop from + its history", or "run a loop retro". Not for use inside an active loop + iteration. +version: 1.0.0 +effort: high +allowed-tools: + - Bash + - Read + - Glob + - Grep + - Write + - Edit +--- + +# SDLC Retro + +Every `/sdlc` run archives its full trace (transition history, decisions, signs, +escalations) and a composite score to `~/.claude/autonomous-sdlc/`. This skill +periodically turns that ledger into improvement proposals for the SDLC skill +itself — AFlow's expand-and-keep-if-improved loop, amortized over real usage +(design: `docs/aflow-sdlc-optimization.md` in the marketplace repo). + +**The contract: the retro proposes; the human merges.** You never edit shipped +skill text and self-approve it. Every proposal lands as a reviewable diff (author +mode) or a feedback entry (consumer mode). + +`RETRO=${CLAUDE_PLUGIN_ROOT}/scripts/sdlc_retro.py` (run with `python3`). + +## Procedure + +### 0. Digest + +```bash +python3 $RETRO digest +``` + +If `runs < 5`, report that there isn't enough data for patterns yet (say how many +runs exist and that the ledger grows automatically) and stop — do **not** mark a +retro. Otherwise summarize the digest for the user in 3-5 lines before diving in. + +### 1. Grade the previous retro first + +If `previous_retro` is set, compare `by_plugin_version` windows before vs. after +that retro's `plugin_version`, using the `pessimistic` field and only windows with +`comparable: true`: + +- **Improved or neutral** → note it in your final report. +- **Regressed** → your first proposal is a revert of that retro's change, with + both windows as evidence. +- Either side not `comparable` → say "too early to grade" and move on. + +### 2. Mine the window + +Drill into the `worst_runs` archive directories (each holds the run's +`state.json`, `decisions.jsonl`, `progress.md`, `signs.md`, `escalation.md`). +Work the checklist, roughly in order of value: + +1. **Recurring BLOCKED reasons** — same budget, no-progress, or escalation shape + across runs? +2. **Rework clusters** — which state pairs bounce most (`rework_totals`), and what + the transition `reason` strings have in common. +3. **Attempts-exceeded shapes** — what kinds of tasks burn all their strikes. +4. **Iteration sinks** — chronic `wait_ticks`, states eating iterations without + producing commits. +5. **Sign audit** — compare scores of runs with vs. without each sign + (`signs_active` + the archived `signs.md`). An unhelpful sign becomes a + removal proposal — signs are finally removable. +6. **Decision audit** — decisions that later correlate with REPAIR or rework. + +### 3. Propose (≤5, each grounded) + +Each proposal needs **at least 2 supporting runs**, cited by their `archive` +paths. A single pathological run yields at most a `signs.md` candidate, never a +skill-text proposal. Eligible targets: + +- `sdlc-loop/SKILL.md` dispatch-table prose (the workflow itself) +- default budgets / review-gate defaults in `sdlc_state.py` +- Architect / Builder agent prompts +- sign graduation into skill text, or sign removal +- the score weights in `sdlc_state.py` — allowed, but flagged as changing the + ruler and **never bundled** with proposals measured by it + +### 4. Deliver + +**Author mode** (the plugin source is a git checkout you can edit — e.g. you are +in the marketplace repo, or the user points you at their clone): create a branch, +apply each proposal as **its own commit** (so a later regression is revertible at +proposal granularity), open a PR whose body lists each proposal with its +supporting runs and the previous-retro grade. Follow the repo's versioning rules. + +**Consumer mode** (plugin installed read-only from the marketplace): record each +proposal via the feedback skill +(`python3 ${CLAUDE_PLUGIN_ROOT}/scripts/feedback_manager.py autonomous-sdlc save ...`) +and present the full proposal text to the user so they can file it upstream. + +### 5. Mark + +Only after delivering (PR opened or feedback saved): + +```bash +python3 $RETRO mark --note "" +``` + +This windows the next retro. Never mark a retro that produced nothing — leave the +window open so the data keeps accumulating. + +## Report + +End with: the previous-retro grade, each proposal (one line + supporting-run +count), where they landed (PR URL / feedback entries), and the next step for the +human (review the PR; keep running `/sdlc` to grow the ledger).