From fb1d793121f7967c2e086f0d38a394c02036b71e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 05:40:27 +0000 Subject: [PATCH 1/4] Add design doc: meta-agent search (ADAS) over plugin workflows Applies ADAS-style meta-agent search to the marketplace's own artifacts: skill descriptions (tier 1, scored against evals/*.json), skill procedures (tier 2), and full autonomous-sdlc workflow variants (tier 3). Reuses autoloop's runner/METRIC/ledger conventions with a population archive instead of single-file hill-climbing, and phases the work so the cheap trigger-description search debugs the machinery before any hours-long SDLC benchmark runs. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015Dexou9QTmNb2Vf8XhedgQ --- docs/meta-agent-search-design.md | 210 +++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 docs/meta-agent-search-design.md diff --git a/docs/meta-agent-search-design.md b/docs/meta-agent-search-design.md new file mode 100644 index 0000000..23248cd --- /dev/null +++ b/docs/meta-agent-search-design.md @@ -0,0 +1,210 @@ +# Meta-Agent Search over Plugin Workflows (ADAS applied to oliphant-plugins) + +**Status**: Proposed design (not yet implemented) +**Prompted by**: Automated Design of Agentic Systems (ADAS; Hu et al. 2025) — "meta-agent +search", where a meta-agent proposes new agentic workflow designs, evaluates them, and +archives the winners. Workflow design is a search problem; we should be able to find good +designs by algorithm, not only by hand. + +## 1. Thesis + +Every plugin in this marketplace is a hand-crafted point in an enormous design space: +skill descriptions, skill procedures, agent prompts, state graphs, gate orderings, hook +configurations. We iterate on them by intuition and dogfooding. ADAS says: treat each of +those artifacts as a **genome**, define a **fitness function**, and let a **meta-agent** +run the search. + +This repo is unusually well positioned to try this, because two of the three ADAS +ingredients already exist here: + +| ADAS ingredient | ADAS paper | This repo — already have | +|---|---|---| +| Seed archive of simple agents | CoT, self-refine, debate | Nine hand-crafted plugins; each is a working, non-trivial seed | +| Search loop harness | Bespoke Python driver | `autoloop`: program.md + immutable `auto/run.sh` + `METRIC` contract + git checkpoint/rollback | +| Fitness function | Held-out benchmark accuracy | **Missing** — but `evals/*.json` is 90% of a cheap one | + +The missing piece is small: an eval runner that turns `evals/*.json` into a scalar, and a +meta-agent protocol that proposes candidates instead of hill-climbing a single file. + +## 2. What the genome is here + +ADAS searches over Python programs. Our "programming language" is the Claude Code plugin +format — SKILL.md files, agent frontmatter + prompts, hooks.json, and (for +autonomous-sdlc) the state machine in `sdlc_state.py`. This is a *better* genome than +free-form code in one important way: it is declarative and mechanically gate-checkable +(`check_marketplace_versions.py`, `sync_shared.py`, hooks JSON parsing, `pytest` on the +Python helpers) before any expensive evaluation runs. + +The search spaces, ordered by evaluation cost: + +| Tier | Genome (what the meta-agent may mutate) | Fitness signal | Cost per candidate | +|---|---|---|---| +| **1 — Trigger surface** | The `description` field of one SKILL.md | Trigger accuracy against `evals/-eval.json`, judged by haiku | Seconds, ~cents | +| **2 — Skill procedure** | SKILL.md body + `references/*.md` | Rubric-judged output quality on fixture tasks (e.g. mochi-creator already ships prompt-quality validation criteria) | Minutes | +| **3 — Whole workflow** | autonomous-sdlc: state graph transitions, Architect/Builder prompts, gate order, loop Stop-hook prompt | Task success rate, iterations-to-green, token cost on a benchmark suite of small repos | Hours | + +Structural validity (plugin.json parses, hooks load, Python compiles, versions sync) is +**not** fitness — it is the quality-gate layer, exactly as in autoloop: gates fail fast +so no benchmark time is wasted on a malformed candidate. + +## 3. Architecture + +### 3.1 The archive + +``` +meta/ +├── archive/ +│ ├── index.jsonl # one line per candidate: id, parents, tier, scores +│ └── {candidate-id}/ +│ ├── genome/ # the mutated files (mirror of plugin-relative paths) +│ └── meta.json # idea description, lineage, fitness per eval, +│ # dev vs holdout scores, tokens spent +└── benchmarks/ # tier-3 only: task suite definitions +``` + +Seeded with: +- **Candidate 0**: the current committed workflow, verbatim (the hand-crafted incumbent). +- **2–3 deliberately simple baselines**, analogous to ADAS seeding with CoT/self-refine. + For tier 3: "single pass, no gates", "build once + self-refine once". These anchor the + low end so fitness deltas are interpretable and give the meta-agent structurally + diverse inspiration. + +Following ADAS, every candidate that passes the gates is archived *with its scores* — +selection pressure is applied at proposal time (inspiration is sampled weighted by +fitness), not by discarding losers. Failed ideas are cheap negative examples. + +### 3.2 The meta-agent loop + +One iteration, mapping ADAS steps onto Claude Code mechanics: + +1. **Read** `meta/archive/index.jsonl` + the top-k candidates' `meta.json` ideas. +2. **Propose**: write a high-level prose description of a new workflow first. Forced + novelty framing: the description must name which archive entries inspired it and + state, in one sentence, how it differs from *each* of its parents. +3. **Implement**: materialize the idea as files under `meta/archive/{new-id}/genome/`. +4. **Self-refine ×2** (Madaan et al. 2023, as in ADAS): + - Pass A — *novelty check*: compare against the archive; if it is a re-skin of an + existing candidate, revise or abandon. + - Pass B — *feasibility check*: do referenced files exist, do hooks parse, does the + procedure contradict itself, does it violate repo invariants (e.g. VERIFY/REVIEW + must call built-in skills, `/goal` stays user-only)? +5. **Evaluate**: run the immutable runner (gates → fitness eval → `METRIC` lines). +6. **Archive**: append to `index.jsonl` with scores; `git commit` the candidate dir. +7. Repeat until `--max-iterations` or token budget. + +This is autoloop's seven components with two deliberate deviations, which is why it is a +sibling protocol rather than a stock autoloop `program.md`: + +- **Mutable artifact** is not one file edited in place — it is a *new directory per + iteration* (population, not trajectory). Checkpoint/rollback becomes append/skip. +- **Keep/revert** is not metric-gated — gates decide archival, the metric only steers + future sampling. Hill-climbing would converge on one lineage; ADAS's results come from + breadth. + +Everything else carries over verbatim: immutable runner, `METRIC key=value` contract, +`results.tsv` ledger, embedded progress log, tiered fastest-first gates, launch via +`claude --dangerously-skip-permissions -p "Read program.md and execute the loop protocol."`. + +### 3.3 The fitness function (tier 1 — build this first) + +`evals/run_trigger_eval.py`: + +- Input: a SKILL.md path (or raw description) + an `evals/*-eval.json` file. +- For each case, ask a judge model (haiku — cheap, and trigger decisions are made by a + small router-shaped judgment anyway) whether the description would fire for the query, + given a realistic "available skills" context so the judge sees competing descriptions. +- Output: `METRIC trigger_score=` plus secondary metrics. + +Scoring detail that matters: report **false-positive rate separately** and weight it +into the primary metric. An over-triggering skill pollutes every unrelated session; a +under-triggering one costs only its own invocations. Balanced accuracy with an FP +penalty, not raw accuracy. + +This script is worth building even if the meta-agent never ships: it is a regression +test for every description edit we already make by hand, runnable in CI. + +### 3.4 Tier 3: searching the SDLC loop itself + +The genome is a variant overlay on `plugins/autonomous-sdlc/`: state-graph edits +(e.g. "merge REVIEW into VERIFY", "add a CRITIQUE state between PLAN and BUILD", +"REPAIR retries twice before BLOCKED"), Architect/Builder prompt rewrites, gate +reordering, loop-prompt wording. The state machine living in `sdlc_state.py` + +`.sdlc/state.json` is what makes this searchable at all — transitions are data, not +vibes. + +Evaluation: 3–5 benchmark micro-repos under `meta/benchmarks/` (a repo with failing +tests + a spec; a small feature request; a bug with a reproduction). Each candidate runs +the full loop in a disposable git worktree (reuse `hooks/scripts/worktree-create.sh` / +`worktree-remove.sh`), headless, with a hard iteration ceiling and token budget. + +``` +fitness = w1·success_rate − w2·norm(tokens) − w3·norm(wallclock) +secondary (tracked, not optimized): diff size, test count delta, BLOCKED rate +``` + +This is the true ADAS regime and the expensive one — hours per candidate. It is phase 3 +for a reason: the tier-1 pilot debugs the archive/meta-agent/runner machinery for cents +before any candidate costs hours. + +## 4. Goodhart defenses + +The eval sets are small (~10–40 cases each), so overfitting is the default outcome, not +a tail risk. + +1. **Dev/holdout split**: each `evals/*.json` is split once (e.g. 70/30, committed, never + reshuffled). The meta-agent sees and optimizes dev scores only; `run.sh` emits both; + the holdout number is what a human reads when deciding whether a winner is real. +2. **Secondary guardrail metrics**: description token count (tier 1 — blocks the + degenerate "enumerate every trigger phrase" solution), diff size and test-delta + (tier 3 — blocks "delete the hard parts"). +3. **Winners land as PRs, never auto-merge.** The search loop only ever writes under + `meta/`. Promoting a winning genome back into `plugins/` is a human-reviewed PR with + the usual version bump + marketplace sync. The meta-agent proposes; the human ships. +4. **Immutable runner** (autoloop's rule, unchanged): the meta-agent can never touch + `auto/run.sh`, the eval JSONs, or the judge prompt. The optimizer must not hold the + measuring stick. + +## 5. Where it lives + +Recommendation: **repo-level `meta/` + `evals/run_trigger_eval.py`, not a new plugin.** +Like `evals/`, this is tooling *about* this repo's artifacts, not a capability to +distribute — it references our eval fixtures and our plugins in place. Graduate it to a +plugin only if the protocol proves general enough to point at someone else's skills +directory. This also keeps the marketplace catalog untouched (no `metadata.version` +bump, no new sync surface) while the idea is unproven. + +`autoloop` stays as-is. The meta-search `program.md` is written fresh following +autoloop's conventions (same runner contract, same ledger) rather than generated by the +autoloop skill, because of the population-vs-trajectory deviation in §3.2. + +## 6. Phased plan + +| Phase | Deliverable | Exit criterion | +|---|---|---| +| **1 — Fitness function** | `evals/run_trigger_eval.py` + dev/holdout split + baseline scores for all nine current skill descriptions, committed to `meta/archive/` as candidate 0 | Baseline table exists; runner is deterministic enough that re-runs agree within noise | +| **2 — Pilot search** | Meta-search `program.md` + `auto/run.sh`; overnight run on the *single worst-scoring* description from phase 1 | ≥1 candidate beats the incumbent on **holdout**, or we learn the eval is saturated | +| **3 — Procedure search** | Tier-2 rubric evals for one skill (mochi-creator, which already has quality criteria) | Rubric-judged wins that survive human review | +| **4 — SDLC workflow search** | `meta/benchmarks/` suite + worktree-isolated tier-3 runs over autonomous-sdlc variants | A state-graph or prompt variant beats the incumbent on success rate at ≤ token parity | + +Phase 1 is a good afternoon of work and is independently useful. Phase 4 is the headline +but should not be attempted until phases 1–2 have shaken out the archive mechanics. + +## 7. Risks and open questions + +- **Judge ≠ router.** Tier-1 fitness uses haiku to *simulate* Claude Code's skill + triggering. If the simulation diverges from the real router, we optimize the wrong + thing. Mitigation: spot-check winners in a live session before promotion; keep the + judge prompt frozen per search run so scores are comparable. +- **Eval saturation.** Several descriptions may already score near-perfect on their + small eval sets. Phase 1's baseline table tells us where search has headroom; where + there is none, the move is *growing the eval set* (also a candidate meta-agent task, + but adversarially — one agent proposes hard cases, the fitness judge stays frozen). +- **Tier-3 nondeterminism.** Agentic runs are noisy; one benchmark run per candidate + will mis-rank. Budget ≥2 runs per candidate and rank on mean; accept that tier 3 is + a coarse signal. +- **Cost ceiling.** The loop must carry a hard token budget per run (autoloop's time + budget generalizes); tier 3 especially, where a pathological candidate can loop in + REPAIR. The benchmark harness kills a candidate at N iterations and scores it failed. +- **Scope of mutation.** Letting the meta-agent touch `hooks/scripts/*.sh` (tier 3) + means candidate code executes with real permissions during evaluation. Start with + prompts + state graph only; hook-script mutation needs a sandbox story first. From 3cd8bdaac3dd3f4112ae9f185f5fb7d55d1103d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 05:47:18 +0000 Subject: [PATCH 2/4] Fold AFlow (arXiv 2410.10762) mechanisms into meta-agent search design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The linked paper is AFlow (Zhang et al., ICLR 2025), a sibling of ADAS that searches code-represented workflows with MCTS. Corrects the citation to cover both papers and adopts AFlow's sample-efficiency mechanisms: a per-tier operator library constraining mutations, experience backpropagation in each candidate's meta.json, soft uniform+score-weighted selection, convergence-based early stop, ~5x evaluation runs, and dollar-weighted cost in the tier-3 fitness — with 'run Architect/Builder on cheaper models at equal success rate' promoted to the headline tier-3 objective, mirroring AFlow's cost result. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015Dexou9QTmNb2Vf8XhedgQ --- docs/meta-agent-search-design.md | 79 ++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 14 deletions(-) diff --git a/docs/meta-agent-search-design.md b/docs/meta-agent-search-design.md index 23248cd..ed9661b 100644 --- a/docs/meta-agent-search-design.md +++ b/docs/meta-agent-search-design.md @@ -1,10 +1,24 @@ # Meta-Agent Search over Plugin Workflows (ADAS applied to oliphant-plugins) **Status**: Proposed design (not yet implemented) -**Prompted by**: Automated Design of Agentic Systems (ADAS; Hu et al. 2025) — "meta-agent -search", where a meta-agent proposes new agentic workflow designs, evaluates them, and -archives the winners. Workflow design is a search problem; we should be able to find good -designs by algorithm, not only by hand. +**Prompted by**: two papers that formulate workflow design as a search problem — +findable by algorithm, not only crafted by hand: + +- **ADAS** — Automated Design of Agentic Systems (Hu et al., arXiv 2408.08435): + "meta-agent search", where a meta-agent proposes new agentic workflow designs in code, + self-refines them for novelty and correctness, evaluates them, and grows an archive of + discoveries that seeds future proposals. +- **AFlow** — Automating Agentic Workflow Generation (Zhang et al., arXiv 2410.10762, + ICLR 2025): reformulates the same problem as MCTS over code-represented workflows + (LLM-invoking nodes connected by edges), made sample-efficient by a small library of + reusable **operators**, per-node **experience backpropagation** ("this modification + helped / didn't"), soft mixed-probability selection, and execution feedback. Headline + result: searched workflows let much cheaper models beat GPT-4o at ~4.5% of its + inference cost. + +This design takes ADAS's outer shape (archive + meta-agent + self-refine) and grafts on +AFlow's sample-efficiency mechanisms (operators, experience backprop, convergence stop), +noted inline where each applies. ## 1. Thesis @@ -58,7 +72,9 @@ meta/ │ └── {candidate-id}/ │ ├── genome/ # the mutated files (mirror of plugin-relative paths) │ └── meta.json # idea description, lineage, fitness per eval, -│ # dev vs holdout scores, tokens spent +│ # dev vs holdout scores, tokens spent, and the +│ # experience record: which operator/modification +│ # was applied to the parent and the fitness delta └── benchmarks/ # tier-3 only: task suite definitions ``` @@ -70,8 +86,30 @@ Seeded with: diverse inspiration. Following ADAS, every candidate that passes the gates is archived *with its scores* — -selection pressure is applied at proposal time (inspiration is sampled weighted by -fitness), not by discarding losers. Failed ideas are cheap negative examples. +selection pressure is applied at proposal time, not by discarding losers. Failed ideas +are cheap negative examples. Selection uses AFlow's soft mix of a uniform distribution +and score-weighted probability, rather than pure fitness-weighting — pure exploitation +collapses the search onto one lineage early, which is exactly what the population +buys us over autoloop's hill-climbing. + +### 3.1a Operators: constraining the mutation space (from AFlow) + +Free-form mutation wastes most samples on noise. AFlow's answer is a small library of +named operators — meaningful, composable moves — and the optimizer mostly picks and +wires operators rather than inventing from scratch. Per tier, ours would look like: + +| Tier | Example operators | +|---|---| +| 1 — descriptions | `add-trigger-phrases`, `add-negative-scope` ("do NOT use for…"), `state-semantic-scope` (describe the job, not keywords), `tighten-persona`, `shorten` | +| 3 — SDLC workflow | `insert-state` (e.g. CRITIQUE between PLAN and BUILD), `merge-states`, `add-retry-edge` (REPAIR retries N before BLOCKED), `ensemble-verify` (run VERIFY twice, require agreement), `reorder-gates`, `rewrite-agent-prompt`, `downgrade-model` (see §3.4) | + +Each candidate's `meta.json` records which operator(s) it applied to its parent and the +resulting fitness delta — AFlow's experience backpropagation. The proposal step reads +its parents' experience records, so "adding trigger phrases stopped helping three +generations ago" is visible context, not rediscovered by burning evaluations. The +meta-agent may still propose an off-library mutation (ADAS's open-endedness), but must +label it as such; operators that repeatedly produce positive deltas earn a place in the +library. ### 3.2 The meta-agent loop @@ -90,7 +128,10 @@ One iteration, mapping ADAS steps onto Claude Code mechanics: must call built-in skills, `/goal` stays user-only)? 5. **Evaluate**: run the immutable runner (gates → fitness eval → `METRIC` lines). 6. **Archive**: append to `index.jsonl` with scores; `git commit` the candidate dir. -7. Repeat until `--max-iterations` or token budget. +7. Repeat until `--max-iterations`, the token budget, or convergence — AFlow-style + early stop when the top-k of the archive hasn't changed for N consecutive + iterations. On a small eval set, a search that has stopped finding wins is done, + not warming up. This is autoloop's seven components with two deliberate deviations, which is why it is a sibling protocol rather than a stock autoloop `program.md`: @@ -138,13 +179,22 @@ the full loop in a disposable git worktree (reuse `hooks/scripts/worktree-create `worktree-remove.sh`), headless, with a hard iteration ceiling and token budget. ``` -fitness = w1·success_rate − w2·norm(tokens) − w3·norm(wallclock) +fitness = w1·success_rate − w2·norm(cost) − w3·norm(wallclock) secondary (tracked, not optimized): diff size, test count delta, BLOCKED rate ``` -This is the true ADAS regime and the expensive one — hours per candidate. It is phase 3 -for a reason: the tier-1 pilot debugs the archive/meta-agent/runner machinery for cents -before any candidate costs hours. +Note `cost`, not `tokens`: AFlow's most striking result is not raw accuracy but +accuracy-per-dollar — searched workflows let far cheaper models beat GPT-4o at ~4.5% +of its cost. That translates here into a concrete, falsifiable search objective: +**Architect and Builder are both pinned to Opus today. Can a searched workflow variant +running them on Sonnet (or Haiku Builder + Sonnet Architect) match the incumbent's +success rate?** `downgrade-model` is therefore a first-class operator, and cost is +dollar-weighted so a Sonnet win at equal success rate scores strictly higher than the +Opus incumbent. This is the single most likely place the search pays for itself. + +This is the true meta-search regime and the expensive one — hours per candidate. It is +phase 3 for a reason: the tier-1 pilot debugs the archive/meta-agent/runner machinery +for cents before any candidate costs hours. ## 4. Goodhart defenses @@ -200,8 +250,9 @@ but should not be attempted until phases 1–2 have shaken out the archive mecha there is none, the move is *growing the eval set* (also a candidate meta-agent task, but adversarially — one agent proposes hard cases, the fitness judge stays frozen). - **Tier-3 nondeterminism.** Agentic runs are noisy; one benchmark run per candidate - will mis-rank. Budget ≥2 runs per candidate and rank on mean; accept that tier 3 is - a coarse signal. + will mis-rank. AFlow evaluates each candidate ~5 times on validation; we should + budget ≥2–3 runs per candidate and rank on mean, and accept that tier 3 is a coarse + signal either way. - **Cost ceiling.** The loop must carry a hard token budget per run (autoloop's time budget generalizes); tier 3 especially, where a pathological candidate can loop in REPAIR. The benchmark harness kills a candidate at N iterations and scores it failed. From 00f976d26dfdaa064248d3248e4acf9f76ab455b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 9 Jul 2026 15:54:32 +0000 Subject: [PATCH 3/4] Implement phase 1 of meta-agent search: trigger-eval fitness function MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit evals/run_trigger_eval.py scores a skill description against its evals/*-eval.json fixture via a batched haiku judge (claude CLI), with a deterministic hash-based dev/holdout split, balanced accuracy as the primary metric, false-positive rate reported separately, and autoloop-convention METRIC output. Supports --description-file so candidate genomes score without touching plugins/. meta/ holds the search harness: program.md (loop protocol with operators, soft-mixed parent selection, two self-refine passes, experience records, convergence stop) and the immutable auto/run.sh candidate runner. verification-stack-eval.json is deliberately unmapped — it targeted the skill removed in the v2 loop redesign. Tests use a stub judge only (no network); pyproject testpaths now includes evals/ so they run in CI. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015Dexou9QTmNb2Vf8XhedgQ --- evals/run_trigger_eval.py | 276 +++++++++++++++++++++++++++++++++ evals/test_run_trigger_eval.py | 105 +++++++++++++ meta/README.md | 33 ++++ meta/auto/run.sh | 29 ++++ meta/program.md | 86 ++++++++++ pyproject.toml | 2 +- 6 files changed, 530 insertions(+), 1 deletion(-) create mode 100644 evals/run_trigger_eval.py create mode 100644 evals/test_run_trigger_eval.py create mode 100644 meta/README.md create mode 100755 meta/auto/run.sh create mode 100644 meta/program.md diff --git a/evals/run_trigger_eval.py b/evals/run_trigger_eval.py new file mode 100644 index 0000000..f10ff3b --- /dev/null +++ b/evals/run_trigger_eval.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python3 +# ABOUTME: Trigger-eval fitness function for meta-agent search (docs/meta-agent-search-design.md). +# ABOUTME: Scores a skill description against evals/*-eval.json via an LLM judge, emits METRIC lines. +"""Score skill descriptions against the trigger-eval fixtures. + +For each ``{"query": ..., "should_trigger": bool}`` case, an LLM judge (haiku via the +``claude`` CLI) decides whether Claude Code would invoke the skill given only its +name and description. Cases are deterministically split into **dev** (optimized by +the meta-search loop) and **holdout** (read by humans judging whether a win is real); +the primary metric is balanced accuracy, with the false-positive rate reported +separately because an over-triggering skill pollutes unrelated sessions. + +Usage: + python evals/run_trigger_eval.py --skill tdd-workflow # one skill + python evals/run_trigger_eval.py --all # every mapped skill + python evals/run_trigger_eval.py --skill tdd-workflow \ + --description-file candidate/SKILL.md # candidate genome + python evals/run_trigger_eval.py --all --judge stub # plumbing test, no LLM + +Output: human-readable table plus autoloop-convention ``METRIC key=value`` lines. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import subprocess +import sys +import tempfile +from dataclasses import dataclass +from pathlib import Path +from typing import Callable, Sequence + +REPO_ROOT = Path(__file__).resolve().parent.parent +EVALS_DIR = REPO_ROOT / "evals" + +# Maps each eval fixture to the SKILL.md whose description it exercises. +# NOTE: verification-stack-eval.json is deliberately absent — it targeted the +# verification-stack skill that autonomous-sdlc v2 removed in favor of the +# built-in /verify (docs/sdlc-loop-redesign.md §1.4). Kept as a fixture only. +EVAL_TO_SKILL: dict[str, str] = { + "bdd-generate": "plugins/autonomous-sdlc/skills/bdd-generate/SKILL.md", + "bdd-spec": "plugins/autonomous-sdlc/skills/bdd-spec/SKILL.md", + "beads-workflow": "plugins/autonomous-sdlc/skills/beads-workflow/SKILL.md", + "tdd-workflow": "plugins/autonomous-sdlc/skills/tdd-workflow/SKILL.md", + "compound-capture": "plugins/compound-knowledge/skills/compound-capture/SKILL.md", + "compound-retrieve": "plugins/compound-knowledge/skills/compound-retrieve/SKILL.md", + "hexagonal-agents": "plugins/hexagonal-agents/skills/hexagonal-agents/SKILL.md", + "mochi-creator": "plugins/mochi-creator/skills/mochi-creator/SKILL.md", +} + +DEV_FRACTION = 0.7 +JUDGE_MODEL = "claude-haiku-4-5-20251001" +JUDGE_BATCH_SIZE = 10 + +# Frozen judge prompt: the optimizer must never hold the measuring stick, so any +# edit to this template invalidates cross-run score comparisons. +JUDGE_PROMPT_TEMPLATE = """\ +You simulate Claude Code's skill-triggering decision. + +A skill is available: +name: {name} +description: {description} + +For each numbered user request below, decide whether Claude Code should invoke this +skill for that request. Judge strictly by the description; many requests are +deliberately adjacent but out of scope. + +Requests: +{numbered_queries} + +Reply with ONLY a JSON array of {n} booleans, one per request, in order. No other text. +""" + +Judge = Callable[[str, str, Sequence[str]], list[bool]] + + +@dataclass +class SplitScores: + n: int + balanced_accuracy: float + tpr: float + tnr: float + fp_rate: float + + +def parse_frontmatter_description(skill_md: str) -> tuple[str, str]: + """Return (name, description) from a SKILL.md's YAML frontmatter. + + Handles the two styles used in this repo: plain scalars and ``>``/``>-`` + folded blocks. Avoids a YAML dependency so the script runs bare. + """ + m = re.match(r"\A---\n(.*?)\n---", skill_md, re.S) + if not m: + raise ValueError("no YAML frontmatter found") + lines = m.group(1).split("\n") + fields: dict[str, str] = {} + key = None + buf: list[str] = [] + folded = False + for line in lines: + top = re.match(r"^(\w[\w-]*):\s*(.*)$", line) + if top: + if key is not None: + fields[key] = " ".join(buf).strip() + key, rest = top.group(1), top.group(2) + folded = rest in {">", ">-", "|", "|-"} + buf = [] if folded else [rest] + elif key is not None and (line.startswith((" ", "\t")) or line == ""): + buf.append(line.strip()) + if key is not None: + fields[key] = " ".join(buf).strip() + if "name" not in fields or "description" not in fields: + raise ValueError("frontmatter missing name or description") + return fields["name"], fields["description"] + + +def split_cases(cases: list[dict]) -> tuple[list[dict], list[dict]]: + """Deterministic dev/holdout split keyed on the query text. + + Hash-based so the split never depends on file order and survives cases being + appended; committed nowhere because it is a pure function of the data. + """ + dev, holdout = [], [] + for case in cases: + digest = hashlib.sha256(case["query"].encode()).digest() + (dev if digest[0] / 256 < DEV_FRACTION else holdout).append(case) + return dev, holdout + + +def score_split(cases: list[dict], verdicts: list[bool]) -> SplitScores: + tp = fn = tn = fp = 0 + for case, verdict in zip(cases, verdicts, strict=True): + if case["should_trigger"]: + tp, fn = tp + int(verdict), fn + int(not verdict) + else: + tn, fp = tn + int(not verdict), fp + int(verdict) + tpr = tp / (tp + fn) if tp + fn else 1.0 + tnr = tn / (tn + fp) if tn + fp else 1.0 + return SplitScores( + n=len(cases), + balanced_accuracy=(tpr + tnr) / 2, + tpr=tpr, + tnr=tnr, + fp_rate=1.0 - tnr, + ) + + +def parse_judge_reply(reply: str, expected_n: int) -> list[bool]: + """Extract a JSON array of booleans, tolerating code fences and prose.""" + m = re.search(r"\[[^\[\]]*\]", reply, re.S) + if not m: + raise ValueError(f"no JSON array in judge reply: {reply[:200]!r}") + verdicts = json.loads(m.group(0)) + if len(verdicts) != expected_n or not all(isinstance(v, bool) for v in verdicts): + raise ValueError(f"expected {expected_n} booleans, got: {verdicts!r}") + return verdicts + + +def claude_cli_judge(name: str, description: str, queries: Sequence[str]) -> list[bool]: + """Judge a batch of queries with one headless haiku call via the claude CLI. + + Runs in a temp cwd so the nested session loads no CLAUDE.md/skills from this + repo. Retries once on unparseable output. + """ + numbered = "\n".join(f"{i + 1}. {q}" for i, q in enumerate(queries)) + prompt = JUDGE_PROMPT_TEMPLATE.format( + name=name, description=description, numbered_queries=numbered, n=len(queries) + ) + last_error: Exception | None = None + for _ in range(2): + result = subprocess.run( + ["claude", "-p", prompt, "--model", JUDGE_MODEL, "--max-turns", "1"], + capture_output=True, + text=True, + timeout=300, + cwd=tempfile.gettempdir(), + ) + try: + return parse_judge_reply(result.stdout, len(queries)) + except (ValueError, json.JSONDecodeError) as e: + last_error = e + raise RuntimeError(f"judge failed twice for {name}: {last_error}") + + +def stub_judge(name: str, description: str, queries: Sequence[str]) -> list[bool]: + """No-LLM judge for plumbing tests: triggers iff the skill name appears verbatim.""" + return [name.lower() in q.lower() for q in queries] + + +def judge_all(name: str, description: str, cases: list[dict], judge: Judge) -> list[bool]: + verdicts: list[bool] = [] + for i in range(0, len(cases), JUDGE_BATCH_SIZE): + batch = cases[i : i + JUDGE_BATCH_SIZE] + verdicts.extend(judge(name, description, [c["query"] for c in batch])) + return verdicts + + +def evaluate_skill( + eval_name: str, + description_file: Path, + judge: Judge, + eval_file: Path | None = None, +) -> dict: + """Evaluate one description against one eval fixture. Returns a result dict.""" + eval_path = eval_file or EVALS_DIR / f"{eval_name}-eval.json" + cases = json.loads(eval_path.read_text()) + name, description = parse_frontmatter_description(description_file.read_text()) + dev, holdout = split_cases(cases) + results = {} + for split_name, split in (("dev", dev), ("holdout", holdout)): + verdicts = judge_all(name, description, split, judge) + results[split_name] = score_split(split, verdicts) + return { + "eval": eval_name, + "skill_name": name, + "description_file": str(description_file), + "description_words": len(description.split()), + "dev": vars(results["dev"]), + "holdout": vars(results["holdout"]), + } + + +def emit(result: dict, metric_prefix: str = "") -> None: + d, h = result["dev"], result["holdout"] + print( + f"{result['eval']:>20} dev {d['balanced_accuracy']:.3f} " + f"(tpr {d['tpr']:.2f} fp {d['fp_rate']:.2f} n={d['n']}) " + f"holdout {h['balanced_accuracy']:.3f} " + f"(tpr {h['tpr']:.2f} fp {h['fp_rate']:.2f} n={h['n']}) " + f"desc {result['description_words']}w" + ) + p = f"{metric_prefix}{result['eval']}_" + print(f"METRIC {p}dev_score={d['balanced_accuracy']:.4f}") + print(f"METRIC {p}holdout_score={h['balanced_accuracy']:.4f}") + print(f"METRIC {p}dev_fp_rate={d['fp_rate']:.4f}") + print(f"METRIC {p}desc_words={result['description_words']}") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--skill", help="eval name from the mapping (e.g. tdd-workflow)") + parser.add_argument("--all", action="store_true", help="run every mapped skill") + parser.add_argument( + "--description-file", + type=Path, + help="SKILL.md to score instead of the committed one (candidate genomes)", + ) + parser.add_argument("--judge", choices=["haiku", "stub"], default="haiku") + parser.add_argument("--json", action="store_true", help="also dump full results as JSON") + args = parser.parse_args(argv) + + if bool(args.skill) == args.all: + parser.error("exactly one of --skill or --all is required") + if args.all and args.description_file: + parser.error("--description-file only makes sense with --skill") + + judge = claude_cli_judge if args.judge == "haiku" else stub_judge + targets = list(EVAL_TO_SKILL) if args.all else [args.skill] + results = [] + for eval_name in targets: + if eval_name not in EVAL_TO_SKILL: + parser.error(f"unknown skill {eval_name!r}; known: {', '.join(EVAL_TO_SKILL)}") + description_file = args.description_file or REPO_ROOT / EVAL_TO_SKILL[eval_name] + results.append(evaluate_skill(eval_name, description_file, judge)) + emit(results[-1]) + + if args.json: + print(json.dumps(results, indent=2)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/evals/test_run_trigger_eval.py b/evals/test_run_trigger_eval.py new file mode 100644 index 0000000..55411dd --- /dev/null +++ b/evals/test_run_trigger_eval.py @@ -0,0 +1,105 @@ +# ABOUTME: Unit tests for the trigger-eval fitness function (run_trigger_eval.py). +# ABOUTME: Uses the stub judge only — no network, no claude CLI, safe for CI. +from pathlib import Path + +import pytest +from run_trigger_eval import ( + EVAL_TO_SKILL, + EVALS_DIR, + REPO_ROOT, + evaluate_skill, + parse_frontmatter_description, + parse_judge_reply, + score_split, + split_cases, + stub_judge, +) + +PLAIN_SKILL = """--- +name: my-skill +description: Does a thing when asked. +--- +body +""" + +FOLDED_SKILL = """--- +name: my-skill +description: >- + Does a thing + across lines. +allowed-tools: [Read] +--- +body +""" + + +def test_parse_plain_description(): + name, desc = parse_frontmatter_description(PLAIN_SKILL) + assert (name, desc) == ("my-skill", "Does a thing when asked.") + + +def test_parse_folded_description(): + name, desc = parse_frontmatter_description(FOLDED_SKILL) + assert (name, desc) == ("my-skill", "Does a thing across lines.") + + +def test_parse_rejects_missing_frontmatter(): + with pytest.raises(ValueError): + parse_frontmatter_description("just a body") + + +def test_every_mapped_skill_parses(): + for eval_name, rel_path in EVAL_TO_SKILL.items(): + skill_md = (REPO_ROOT / rel_path).read_text() + name, desc = parse_frontmatter_description(skill_md) + assert desc, f"{eval_name}: empty description" + assert (EVALS_DIR / f"{eval_name}-eval.json").exists() + + +def test_split_is_deterministic_and_partitions(): + cases = [{"query": f"query number {i}", "should_trigger": i % 2 == 0} for i in range(40)] + dev1, holdout1 = split_cases(cases) + dev2, holdout2 = split_cases(list(reversed(cases))) + assert {c["query"] for c in dev1} == {c["query"] for c in dev2} + assert len(dev1) + len(holdout1) == len(cases) + assert dev1 and holdout1 # both splits non-empty at this size + + +def test_score_split_math(): + cases = [ + {"query": "a", "should_trigger": True}, + {"query": "b", "should_trigger": True}, + {"query": "c", "should_trigger": False}, + {"query": "d", "should_trigger": False}, + ] + # one miss, one false positive + scores = score_split(cases, [True, False, True, False]) + assert scores.tpr == 0.5 + assert scores.tnr == 0.5 + assert scores.fp_rate == 0.5 + assert scores.balanced_accuracy == 0.5 + + +def test_parse_judge_reply_strips_fences_and_prose(): + reply = 'Sure!\n```json\n[true, false, true]\n```\n' + assert parse_judge_reply(reply, 3) == [True, False, True] + + +def test_parse_judge_reply_rejects_wrong_arity(): + with pytest.raises(ValueError): + parse_judge_reply("[true, false]", 3) + + +def test_evaluate_skill_end_to_end_with_stub(tmp_path: Path): + eval_file = tmp_path / "fake-eval.json" + eval_file.write_text( + '[{"query": "use my-skill please", "should_trigger": true},' + ' {"query": "unrelated request", "should_trigger": false}]' + ) + skill_file = tmp_path / "SKILL.md" + skill_file.write_text(PLAIN_SKILL) + result = evaluate_skill("bdd-spec", skill_file, stub_judge, eval_file=eval_file) + assert result["skill_name"] == "my-skill" + assert result["dev"]["n"] + result["holdout"]["n"] == 2 + for split in ("dev", "holdout"): + assert 0.0 <= result[split]["balanced_accuracy"] <= 1.0 diff --git a/meta/README.md b/meta/README.md new file mode 100644 index 0000000..70a7c46 --- /dev/null +++ b/meta/README.md @@ -0,0 +1,33 @@ +# meta/ + +Proof-of-concept of ADAS/AFlow-style meta-agent search over this repo's own workflow +artifacts. Design and rationale: `docs/meta-agent-search-design.md`. Like `evals/`, +this is repo-level tooling, not a plugin. + +| Piece | Role | +|---|---| +| `program.md` | The meta-agent loop protocol (tier 1: skill trigger descriptions) | +| `auto/run.sh` | **Immutable** candidate runner: gates → fitness → `METRIC` lines | +| `archive/index.jsonl` | Ledger: one line per candidate with scores | +| `archive/candidate-NNN/` | One candidate: `genome/` (mutated files) + `meta.json` (idea, lineage, operator, scores, experience) | + +Candidate-000 is the hand-crafted incumbent — its genome is the committed tree at the +recorded SHA, and its scores are the baseline every search run climbs from. + +The fitness function is `evals/run_trigger_eval.py` (haiku judge via the `claude` CLI, +deterministic dev/holdout split, balanced accuracy + separate false-positive rate). +It is also useful standalone as a regression check when hand-editing a description: + +```bash +python3 evals/run_trigger_eval.py --skill mochi-creator +python3 evals/run_trigger_eval.py --all +``` + +Launch a search run: + +```bash +claude --dangerously-skip-permissions -p "Read meta/program.md and execute the loop protocol." +``` + +Winners are promoted to `plugins/` only via human-reviewed PR — the loop itself never +touches anything outside `meta/archive/`. diff --git a/meta/auto/run.sh b/meta/auto/run.sh new file mode 100755 index 0000000..06fcef6 --- /dev/null +++ b/meta/auto/run.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# ABOUTME: Immutable evaluation runner for meta-agent search candidates (tier 1). +# ABOUTME: Gates a candidate's genome, then scores each mutated SKILL.md — emits METRIC lines. +# +# Usage: meta/auto/run.sh +# +# A candidate dir contains genome/ mirroring plugin-relative paths, e.g. +# meta/archive/candidate-001/genome/plugins/mochi-creator/skills/mochi-creator/SKILL.md +# Only mutated files are present. The meta-agent must NEVER edit this script, +# evals/*.json, or the judge prompt in evals/run_trigger_eval.py. +set -euo pipefail +cd "$(dirname "$0")/../.." + +CANDIDATE_DIR="${1:?usage: meta/auto/run.sh }" + +# Gate 1: candidate structure — a genome with at least one SKILL.md. +mapfile -t SKILL_FILES < <(find "$CANDIDATE_DIR/genome" -name SKILL.md 2>/dev/null | sort) +if [[ ${#SKILL_FILES[@]} -eq 0 ]]; then + echo "GATE FAIL: no SKILL.md under $CANDIDATE_DIR/genome" >&2 + exit 1 +fi + +# Gate 2 + fitness: frontmatter must parse and score against the skill's eval +# fixture (run_trigger_eval.py exits non-zero on unparseable frontmatter or an +# unmapped skill, so a malformed genome fails fast before judge spend). +for skill_md in "${SKILL_FILES[@]}"; do + eval_name="$(basename "$(dirname "$skill_md")")" + python3 evals/run_trigger_eval.py --skill "$eval_name" --description-file "$skill_md" +done diff --git a/meta/program.md b/meta/program.md new file mode 100644 index 0000000..477c7c9 --- /dev/null +++ b/meta/program.md @@ -0,0 +1,86 @@ +# Meta-Agent Search — Tier 1: Skill Trigger Descriptions + +You are the meta-agent in an ADAS/AFlow-style search over this repo's skill +descriptions (design: `docs/meta-agent-search-design.md`). Each iteration you propose +a new candidate description for a target skill, evaluate it against its trigger-eval +fixture, and archive the result with scores and an experience record. You optimize +**dev** scores only; holdout is recorded but must never influence your proposals. + +## Hard rules + +- **Never edit**: `meta/auto/run.sh`, `evals/*.json`, `evals/run_trigger_eval.py` + (especially the judge prompt), or any file under `plugins/`. You write only inside + `meta/archive/`. +- One candidate per iteration. Stop after **10 iterations**, or earlier if the best + dev score hasn't improved for **3 consecutive candidates** (converged), or if the + incumbent dev score for the target skill is already 1.0 (saturated — say so and stop). +- Log every candidate to the archive, including failures — they are negative examples. + +## Target selection + +Read `meta/archive/index.jsonl`. Unless the launcher named a skill, pick the skill +with the **lowest dev balanced accuracy** in candidate-000 that is not yet saturated. +All iterations in one run target the same skill. + +## Iteration protocol + +1. **Read the archive.** `meta/archive/index.jsonl`, then the `meta.json` of every + candidate for the target skill. Note each candidate's operator, fitness delta vs + its parent, and idea text. Operators that repeatedly produced negative deltas are + exhausted — say so and avoid them. +2. **Select a parent.** Flip a coin (conceptually): half the time take the highest + dev-scoring candidate for the skill, half the time pick uniformly among the others. + This soft mix keeps exploration alive. +3. **Propose (idea first).** Write 2–4 sentences: which parent, which operator(s), + and how the result differs from every prior candidate for this skill. Operators: + + | Operator | Move | + |---|---| + | `add-trigger-phrases` | add concrete quoted user phrasings that should fire | + | `add-negative-scope` | add explicit "do NOT use for …" boundaries | + | `state-semantic-scope` | describe the *job* the skill does, not keywords | + | `tighten-persona` | remove hedges/marketing; imperative MUST/use-when framing | + | `shorten` | cut redundant clauses; guardrail vs description bloat | + | `off-library:` | anything else — label it and describe the move | + +4. **Self-refine ×2 before evaluating.** + - *Novelty pass*: compare the draft against every archived candidate for this + skill. If it is a re-skin of one of them, revise or pick a different operator. + - *Feasibility pass*: frontmatter must stay valid YAML with only the + `description` changed; the description must stay truthful to what the skill + actually does (read the skill body if unsure — a description that promises + behavior the skill lacks is a lie that scores well). +5. **Materialize.** Next id `candidate-NNN`. Copy the parent's SKILL.md into + `meta/archive/candidate-NNN/genome/` and edit only the + description. +6. **Evaluate.** `./meta/auto/run.sh meta/archive/candidate-NNN` — if a gate fails, + fix the genome once; if it fails again, archive it as `"gate_failed": true` and + move on. +7. **Archive.** Write `meta/archive/candidate-NNN/meta.json`: + + ```json + { + "id": "candidate-NNN", + "skill": "", + "parents": ["candidate-PPP"], + "operator": "", + "idea": "", + "scores": {"dev": ..., "holdout": ..., "dev_fp_rate": ..., "desc_words": ...}, + "delta_vs_parent": {"dev": ...}, + "experience": "" + } + ``` + + Append one line to `meta/archive/index.jsonl` with id, skill, operator, parents, + dev, holdout. `git add meta/archive && git commit -m "meta: candidate-NNN dev="`. +8. **Update the progress log** below (one bullet per iteration), then loop. + +## Ending the run + +Report: best candidate by **dev** score, its **holdout** score alongside the +incumbent's, and each operator's aggregate delta. Do **not** edit anything under +`plugins/` — promoting a winner is a human-reviewed PR (design doc §4.3). + +## Progress log + +- (empty — appended by the meta-agent, one line per iteration) diff --git a/pyproject.toml b/pyproject.toml index b306a01..ccd99d4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,7 +19,7 @@ dev = ["pytest>=8", "ruff>=0.6"] # review-diff's coverage-gated config lives in its own pyproject and only applies # when pytest is invoked from that directory; at the repo root those tests still # run (without the coverage gate), so nothing is excluded here. -testpaths = ["plugins"] +testpaths = ["plugins", "evals"] [tool.ruff] line-length = 100 From fb783f0785f40de955a85015b1f1499fd250ff4a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 9 Jul 2026 16:01:10 +0000 Subject: [PATCH 4/4] Run first meta-search iteration: beads-workflow description, dev .833 -> 1.0 Baseline (candidate-000) scores all 8 mapped descriptions: 4 already saturated at dev 1.0, headroom on beads-workflow (.833), compound-retrieve (.850), bdd-generate (.917), mochi-creator (.929). Candidate-001 applies add-trigger-phrases to beads-workflow, adding the bd lifecycle surfaces the incumbent omitted (stats/project health, sync/dolt remote sharing, ready/unblocked work). Result: dev .833 -> 1.0 AND holdout .875 -> 1.0 with zero false positives, at the cost of a 56 -> 99 word description. Archived with lineage, operator, and experience record; design doc gains a section 8 with PoC results. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_015Dexou9QTmNb2Vf8XhedgQ --- docs/meta-agent-search-design.md | 32 +++- meta/archive/candidate-000/meta.json | 63 ++++++++ .../skills/beads-workflow/SKILL.md | 146 ++++++++++++++++++ meta/archive/candidate-001/meta.json | 21 +++ meta/archive/index.jsonl | 2 + 5 files changed, 263 insertions(+), 1 deletion(-) create mode 100644 meta/archive/candidate-000/meta.json create mode 100644 meta/archive/candidate-001/genome/plugins/autonomous-sdlc/skills/beads-workflow/SKILL.md create mode 100644 meta/archive/candidate-001/meta.json create mode 100644 meta/archive/index.jsonl diff --git a/docs/meta-agent-search-design.md b/docs/meta-agent-search-design.md index ed9661b..0112bb9 100644 --- a/docs/meta-agent-search-design.md +++ b/docs/meta-agent-search-design.md @@ -1,6 +1,7 @@ # Meta-Agent Search over Plugin Workflows (ADAS applied to oliphant-plugins) -**Status**: Proposed design (not yet implemented) +**Status**: Phases 1–2 implemented as a proof of concept (`evals/run_trigger_eval.py` + +`meta/`); see §8 for first results. Phases 3–4 remain proposed. **Prompted by**: two papers that formulate workflow design as a search problem — findable by algorithm, not only crafted by hand: @@ -259,3 +260,32 @@ but should not be attempted until phases 1–2 have shaken out the archive mecha - **Scope of mutation.** Letting the meta-agent touch `hooks/scripts/*.sh` (tier 3) means candidate code executes with real permissions during evaluation. Start with prompts + state graph only; hook-script mutation needs a sandbox story first. + +## 8. Proof-of-concept results (phases 1–2) + +Implemented: `evals/run_trigger_eval.py` (haiku judge via the `claude` CLI, batched; +deterministic hash-based dev/holdout split; balanced accuracy + separate FP rate; +`METRIC` output) and the `meta/` harness (`program.md` loop protocol, immutable +`auto/run.sh`, archive). One search iteration was run by hand to validate the loop +end-to-end. Notes: + +- **`verification-stack-eval.json` is orphaned** — it targeted the skill removed by + the v2 loop redesign, so it is unmapped (8 of 9 fixtures map to live skills). +- **Baseline (candidate-000)**: 4 of 8 descriptions are already saturated at dev 1.0 + (bdd-spec, tdd-workflow, compound-capture, hexagonal-agents) — for those, the next + move is growing the eval sets, not search. Headroom: beads-workflow .833, + compound-retrieve .850, bdd-generate .917, mochi-creator .929 (the only skill with + dev false positives — over-triggering, the costlier failure mode). +- **First searched candidate (candidate-001)**: target beads-workflow, operator + `add-trigger-phrases`. The incumbent covered issue CRUD but not the rest of the bd + lifecycle; adding the missing surfaces (bd stats / project health, bd sync / dolt + remote sharing, ready/unblocked-work phrasing) took **dev .833 → 1.0 and holdout + .875 → 1.0** with FP rate still 0. The holdout jump says the win is real, not dev + overfit. Guardrail cost: description grew 56 → 99 words. +- Judge cost/latency: ~6–15 s per batched haiku call, 2–3 calls per skill per + evaluation — a full 8-skill baseline in ~5 minutes, one candidate in ~30 s. + +Caveats, per §7: the eval sets are small (20 cases), the judge simulates rather than +reproduces the real trigger router, and a single 20-case fixture saturates quickly — +candidate-001 should be spot-checked in a live session before being promoted to +`plugins/` (which stays a human-reviewed PR, not an auto-merge). diff --git a/meta/archive/candidate-000/meta.json b/meta/archive/candidate-000/meta.json new file mode 100644 index 0000000..4345d86 --- /dev/null +++ b/meta/archive/candidate-000/meta.json @@ -0,0 +1,63 @@ +{ + "id": "candidate-000", + "skill": "*", + "parents": [], + "operator": null, + "idea": "Hand-crafted incumbent: the committed descriptions of every mapped skill. Genome is the repo tree at the recorded SHA rather than copies, to avoid drift.", + "genome": { + "ref": "git", + "sha": "00f976d" + }, + "judge": "claude-haiku-4-5-20251001", + "scores": { + "bdd-generate": { + "dev": 0.9167, + "holdout": 0.875, + "dev_fp_rate": 0.0, + "desc_words": 58 + }, + "bdd-spec": { + "dev": 1.0, + "holdout": 1.0, + "dev_fp_rate": 0.0, + "desc_words": 77 + }, + "beads-workflow": { + "dev": 0.8333, + "holdout": 0.875, + "dev_fp_rate": 0.0, + "desc_words": 56 + }, + "tdd-workflow": { + "dev": 1.0, + "holdout": 1.0, + "dev_fp_rate": 0.0, + "desc_words": 48 + }, + "compound-capture": { + "dev": 1.0, + "holdout": 1.0, + "dev_fp_rate": 0.0, + "desc_words": 76 + }, + "compound-retrieve": { + "dev": 0.85, + "holdout": 1.0, + "dev_fp_rate": 0.1, + "desc_words": 72 + }, + "hexagonal-agents": { + "dev": 1.0, + "holdout": 1.0, + "dev_fp_rate": 0.0, + "desc_words": 73 + }, + "mochi-creator": { + "dev": 0.9286, + "holdout": 1.0, + "dev_fp_rate": 0.1429, + "desc_words": 66 + } + }, + "experience": "Baseline. Saturated (dev=1.0): bdd-spec, tdd-workflow, compound-capture, hexagonal-agents. Headroom: beads-workflow .833, compound-retrieve .850, bdd-generate .917, mochi-creator .929 (only one with dev false positives)." +} diff --git a/meta/archive/candidate-001/genome/plugins/autonomous-sdlc/skills/beads-workflow/SKILL.md b/meta/archive/candidate-001/genome/plugins/autonomous-sdlc/skills/beads-workflow/SKILL.md new file mode 100644 index 0000000..8f18fdb --- /dev/null +++ b/meta/archive/candidate-001/genome/plugins/autonomous-sdlc/skills/beads-workflow/SKILL.md @@ -0,0 +1,146 @@ +--- +name: beads-workflow +description: > + MUST use when working with Beads issues via `bd` commands — creating, updating, or closing + issues, checking what's ready to work on, or syncing the issue database. Use proactively when + starting work (bd update --status=in_progress) and completing work (bd close). Trigger: any + mention of "issue", "task tracking", "what should I work on", ready or unblocked work, + "bd create/ready/close/stats/sync", project health or issue statistics (bd stats), pushing or + syncing the beads/dolt issue database to a remote to share with the team (bd sync), or when + the session hook indicates beads is active. Provides full bd CLI reference and workflow patterns. +version: 1.0.0 +effort: low +allowed-tools: + - Bash +--- + +# Beads Workflow for SDLC + +## Goal + +Use the Beads CLI (`bd`) to track work items as git-native files with explicit dependencies. Beads enables autonomous SDLC by making blocking relationships visible — `bd ready` shows what can be worked on now, `bd close` unblocks dependents. + +## Dependencies + +### Tools + +- **`bd` CLI** — Git-native issue tracker. Commands: `bd create`, `bd ready`, `bd close`, `bd dep add`, `bd show`, `bd list`, `bd sync`. +- **Bash** — Runs all `bd` commands. +- **git** — Beads stores work items as files in the repo. + +### Connectors + +- **Git worktrees** (optional) — Each Bead can map to an isolated worktree for parallel execution. + +## Context + +### Core Commands + +**Finding work:** +```bash +bd ready # Tasks with no blockers (ready now) +bd list --status=open # All open tasks +bd list --status=in_progress # Active work +bd show # Full details with dependencies +bd blocked # All blocked tasks +``` + +**Creating work:** +```bash +# Priority: 0=critical, 1=high, 2=medium, 3=low, 4=backlog +# Types: task, feature, bug, epic, chore +bd create --title="Implement feature X" --type=task --priority=1 +``` + +**Managing dependencies:** +```bash +# First arg DEPENDS ON second arg +bd dep add +bd dep remove +``` + +**Completing work:** +```bash +bd close # Unblocks dependents +bd close # Close multiple at once +bd close --reason="Done" # Close with reason +``` + +**Syncing:** +```bash +bd sync # Sync with git remote +bd stats # Project statistics +bd doctor # Check for issues +``` + +### Best Practices + +1. **Granular tasks** — Each Bead should be implementable in one focused session +2. **Clear dependencies** — Use `bd dep add` to make blocking relationships explicit +3. **Close immediately** — Run `bd close` as soon as verification passes +4. **Sync often** — Run `bd sync` after completing work +5. **Check ready first** — Always start with `bd ready` to find unblocked work + +## Process + +### Step 0: Load Stored Feedback + +```bash +python ${CLAUDE_PLUGIN_ROOT}/scripts/feedback_manager.py autonomous-sdlc show-feedback +``` + +Apply relevant feedback: **beads_workflow**, **general**. + +### Step 1: Architect Creates Feature Graph + +Break requirements into Beads with dependencies: + +```bash +bd create --title="Database schema for users" --type=task --priority=1 # → beads-abc +bd create --title="User model and repository" --type=task --priority=1 # → beads-def +bd create --title="Auth middleware" --type=task --priority=1 # → beads-ghi +bd create --title="Login/logout endpoints" --type=feature --priority=1 # → beads-jkl + +# Dependency chain +bd dep add beads-def beads-abc # Model depends on schema +bd dep add beads-ghi beads-def # Middleware depends on model +bd dep add beads-jkl beads-ghi # Endpoints depend on middleware +``` + +### Step 2: Find Ready Work + +```bash +bd ready # Shows beads-abc (no blockers) +``` + +When `bd ready` returns multiple tasks, they can be implemented in parallel (no mutual dependencies). + +### Step 3: Implement and Close + +```bash +# After implementation passes verification +git add -A +git commit -m "feat(beads-abc): implement database schema for users" +bd close beads-abc # Unblocks beads-def +bd sync +``` + +### Step 4: Worktree Integration (Optional) + +Each Bead can map to an isolated worktree. Claude Code provides native worktree isolation via `isolation: "worktree"` on the Task tool — worktree creation and cleanup are automatic: + +```python +# Spawn a builder in an isolated worktree for this Bead +Task( + subagent_type="autonomous-sdlc:builder", + description=f"Build beads-abc", + prompt="Implement the task...", + isolation="worktree", + run_in_background=True +) +# On completion: bd close beads-abc +``` + +## Output + +A dependency graph of Beads tracked as git-native files. The graph drives autonomous workflow: `bd ready` determines what to work on, `bd close` cascades unblocking, and `bd sync` shares progress. diff --git a/meta/archive/candidate-001/meta.json b/meta/archive/candidate-001/meta.json new file mode 100644 index 0000000..3cb3408 --- /dev/null +++ b/meta/archive/candidate-001/meta.json @@ -0,0 +1,21 @@ +{ + "id": "candidate-001", + "skill": "beads-workflow", + "parents": [ + "candidate-000" + ], + "operator": "add-trigger-phrases", + "idea": "Parent candidate-000 (incumbent). The incumbent's trigger list covers create/update/close but not the rest of the bd lifecycle; dev misses cluster on project-health (bd stats) and sharing the issue database (bd sync / dolt remote). Add those surfaces plus 'ready/unblocked work' phrasing, keeping the MUST-use framing. Differs from candidate-000 by covering the full bd command lifecycle rather than only issue CRUD. Feasibility-checked: the skill body documents bd stats and bd sync.", + "judge": "claude-haiku-4-5-20251001", + "scores": { + "dev": 1.0, + "holdout": 1.0, + "dev_fp_rate": 0.0, + "desc_words": 99 + }, + "delta_vs_parent": { + "dev": 0.1667, + "holdout": 0.125 + }, + "experience": "Enumerating the missing lifecycle surfaces (stats, sync/dolt, ready-work) closed all dev misses and generalized to holdout; cost is description growth 56->99 words - a shorten pass is the natural next probe if this were continued." +} diff --git a/meta/archive/index.jsonl b/meta/archive/index.jsonl new file mode 100644 index 0000000..ccf7e25 --- /dev/null +++ b/meta/archive/index.jsonl @@ -0,0 +1,2 @@ +{"id": "candidate-000", "skill": "*", "operator": null, "parents": [], "dev": {"bdd-generate": 0.9167, "bdd-spec": 1.0, "beads-workflow": 0.8333, "tdd-workflow": 1.0, "compound-capture": 1.0, "compound-retrieve": 0.85, "hexagonal-agents": 1.0, "mochi-creator": 0.9286}, "holdout": {"bdd-generate": 0.875, "bdd-spec": 1.0, "beads-workflow": 0.875, "tdd-workflow": 1.0, "compound-capture": 1.0, "compound-retrieve": 1.0, "hexagonal-agents": 1.0, "mochi-creator": 1.0}} +{"id": "candidate-001", "skill": "beads-workflow", "operator": "add-trigger-phrases", "parents": ["candidate-000"], "dev": 1.0, "holdout": 1.0}