autonomous-sdlc: learn from every run — trace capture, scoring, and /sdlc-retro (AFlow-inspired) - #21
Draft
JoshuaOliphant wants to merge 3 commits into
Draft
autonomous-sdlc: learn from every run — trace capture, scoring, and /sdlc-retro (AFlow-inspired)#21JoshuaOliphant wants to merge 3 commits into
JoshuaOliphant wants to merge 3 commits into
Conversation
…loop Maps AFlow (Zhang et al. 2025) onto autonomous-sdlc: what its MCTS workflow search assumes, why a naive port fails, and a three-tier adaptation — score every run from the existing state.json trace, reify the loop's config surface as a hashable workflow genome, then search genome variants with MCTS-lite using autoloop as the benchmark-mode evaluator harness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UT9Qjmu44qeUVEbwfbSfMk
Per discussion: instead of benchmark-driven MCTS as the primary mechanism, every real /sdlc run archives its trace (state history, decisions, signs, escalations) to a durable user-level ledger with a computed score, and a periodic /sdlc-retro mines the accumulated records to propose skill improvements as a human-reviewed PR. Each retro first grades the previous one by comparing score windows across plugin versions (AFlow's keep-if-improved, amortized over real usage). Deliberate genome/benchmark search moves to future work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UT9Qjmu44qeUVEbwfbSfMk
JoshuaOliphant
changed the title
Proof of concept for docs/aflow-sdlc-optimization.md: - sdlc_state.py: every terminal transition (DONE and BLOCKED, including budget force-blocks) archives the run's full trace to the user-level ledger at ~/.claude/autonomous-sdlc/ (override: SDLC_RUNS_DIR) and appends a scored runs.jsonl line carrying plugin_version. New 'score' subcommand computes the composite (outcome, efficiency, rework rate, autonomy) from the current increment's transition history. Archiving is best-effort and can never break a transition. - sdlc_retro.py: 'digest' aggregates the ledger since the last retro marker (per-version score windows with pessimistic means and a min-n comparability gate, outcome/rework totals, worst-run pointers); 'mark' records a retro point. - New sdlc-retro skill + /sdlc-retro command: grade the previous retro, mine worst runs, deliver <=5 grounded proposals (>=2 supporting runs each) as a PR in author mode or feedback entries in consumer mode. - sdlc-loop skill: notes automatic archiving at SHIP->DONE and nudges /sdlc-retro at 10+ unretroed runs. - Tests for scoring, archive-on-terminal, archive-failure safety, increment windowing, and the digest/mark CLI. Plugin 2.3.1 -> 2.4.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UT9Qjmu44qeUVEbwfbSfMk
JoshuaOliphant
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Design note plus proof-of-concept implementation for making the
autonomous-sdlcloop learn from its own runs. Started from AFlow (Zhang et al., ICLR 2025 — MCTS over agentic workflows), revised after discussion to invert the mechanism: instead of buying evaluations with benchmark runs, every real/sdlcrun is the evaluation, amortized over normal usage. Plugin bumped 2.3.1 → 2.4.0.Design (
docs/aflow-sdlc-optimization.md)The flow — capture → score → retro → measure. The retro proposes but never self-merges; patterns need ≥2 supporting runs; secondary metrics veto but are never optimized. The original genome/benchmark MCTS design is retained as future work for when passive accumulation plateaus.
Implementation
sdlc_state.py): every terminal transition (DONE and BLOCKED, including budget force-blocks viablock()) archives the run's full trace (state.jsonhistory,decisions.jsonl,progress.md,signs.md,escalation.md) to~/.claude/autonomous-sdlc/runs/(override:SDLC_RUNS_DIR) and appends a scored line toruns.jsonlcarryingplugin_versionfor attribution. Archiving is best-effort by design — it can never break a transition.sdlc_state.py score): composite from the current increment's transition history — 0.5·outcome + 0.2·efficiency + 0.2·(1−rework rate) + 0.1·autonomy — with rework broken out (verify bounces, review round-trips, replans, repairs).scripts/sdlc_retro.py):digestaggregates runs since the last retro marker — per-version score windows with pessimistic means (mean − stderr) and a min-n comparability gate, outcome/blocked-reason/rework totals, worst-run archive pointers;markrecords the retro point that windows the next one./sdlc-retrocommand +sdlc-retroskill: grade the previous retro first (revert proposal on regression), mine the worst runs against a pattern checklist (blocked reasons, rework clusters, attempts-exceeded shapes, sign audit, decision audit), deliver ≤5 grounded proposals — one commit each, as a PR in author mode or feedback entries in consumer mode.sdlc-loopskill: SHIP→DONE notes the automatic archiving and nudges/sdlc-retroat 10+ unretroed runs.Verification
uv run --group dev python scripts/check_all.py— versions in sync, shared copies match, 131 tests pass (9 new: scoring, archive-on-terminal, forced-block capture, archive-failure safety, increment windowing, digest/mark CLI)ruff check plugins/autonomous-sdlc/scripts/— cleandigestaggregated them and surfaced the blocked run as worst →markwindowed the next digest to 0 runs🤖 Generated with Claude Code
https://claude.ai/code/session_01UT9Qjmu44qeUVEbwfbSfMk