diff --git a/README.md b/README.md index 844ef01..0dd1145 100644 --- a/README.md +++ b/README.md @@ -11,97 +11,115 @@ evaluate โ†’ change โ†’ gate โ†’ repeat ``` -**A template for self-improving Claude Code loops that run unattended without fooling themselves.** +**A template for Claude Code loops that improve something on their own, in small steps, without lying to you about it.** ![CI](https://github.com/mboss37/selfsmith/actions/workflows/ci.yml/badge.svg) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) -**Selfsmith is a copy-and-fill template for building disciplined, unattended, self-improving [Claude Code](https://claude.com/claude-code) loops.** It was extracted from a production harness and hardened so the loop can't game its own metrics and can't wreck your machine. +## In plain terms -You point it at something measurable: an eval score, a latency number, a pass rate. Every iteration, it evaluates the current state, makes **one** reversible change, tries to **kill** that change with an adversarial gate backed by real statistics, and logs the honest outcome, win or kill. Then it does it again. +You give this loop something you can measure: a test pass rate, a latency number, an accuracy score, anything a command can print as a number. Each time it runs, it does four things: + +1. Looks at how things are doing right now. +2. Makes exactly one small, reversible change that it thinks will help. +3. Has a second, skeptical AI try to prove the change is fake, unsafe, or a fluke, using an actual statistics check, not a vibe check. +4. Keeps the change only if it survives that check, and writes down what happened either way, including the times it failed. + +Then it does that again next time you run it. Nothing gets promoted unless a skeptic tried to kill it and couldn't. ```bash git clone https://github.com/mboss37/selfsmith && cd selfsmith/examples/latency-tuner -pip install -r requirements.txt && python -m pytest sim/ -q # watch the whole harness prove itself +pip install -r requirements.txt && python -m pytest sim/ -q # watch the whole thing prove itself claude # then type: /iterate ``` -No API key needed for the examples. They run offline and deterministic. +No API key needed for the examples. They run offline with fake but realistic data. --- ## Table of contents -- [What is this?](#what-is-this) -- [Why not just prompt "improve yourself in a loop"?](#why-not-just-prompt-improve-yourself-in-a-loop) +- [Why this exists](#why-this-exists) +- [How it works](#how-it-works) - [The examples](#the-examples) - [Build your own loop](#build-your-own-loop) -- [How an iteration works](#how-an-iteration-works) +- [How one run works, step by step](#how-one-run-works-step-by-step) - [Ways to run it](#ways-to-run-it) -- [The safety model, before you run unattended](#the-safety-model-before-you-run-unattended) -- [The 10 principles](#the-10-principles) +- [How safe is "unattended"?](#how-safe-is-unattended) +- [The 10 rules](#the-10-rules) - [Repo layout](#repo-layout) +- [Glossary](#glossary) - [FAQ](#faq) - [Contributing](#contributing) --- -## What is this? +## Why this exists -Anthropic's **evaluator-optimizer pattern**, disciplined enough for unattended operation. One role evaluates and diagnoses. Another proposes and implements. A third tries to kill the change. The machine can only improve by running this cycle. It can never promote itself past the gate. +If you just tell an AI "keep improving this in a loop", a few things go wrong in practice, and they go wrong quietly: -```mermaid -flowchart LR - E["๐Ÿ” Evaluator
read-only"] -->|verdict + focus| P["๐Ÿ’ก Proposer
one hypothesis"] - P --> I["๐Ÿ”ง Implementer
one change"] - I --> G{"๐Ÿšซ Gate
default: REJECT
verdict.py is binding"} - G -->|approve| C["โœ… Commit + Log"] - G -->|reject| R["โ†ฉ๏ธ Revert + Log
an honest kill is a win"] - C --> M["๐Ÿ” Meta-improver
may never weaken a safeguard"] - R --> M -``` +- It starts optimizing for the number instead of the thing the number was supposed to measure. A support-triage bot can "improve" by learning a shortcut that works on your test cases and nowhere else. A latency tuner can "improve" by failing fast instead of actually being faster. +- The AI reviewing its own change is the same AI (or the same kind of reasoning) that made the change, so a plausible-sounding but wrong result talks its way through. +- It edits its own safety rules when they get in the way of hitting the number. +- Once there's nothing real left to improve, it manufactures busywork so it looks productive. +- An unattended run gets stuck, runs forever, or does something destructive, and nobody notices until later. -Two properties make it safe to leave running: +Selfsmith is a template that builds a structural answer to each of those, not just a polite instruction asking the AI to please not do that. Here's what backs each one, specifically: -1. **A mechanical floor the loop can't lift.** A shell hook, not an instruction, gates Bash *and* the file-edit tools, so the loop can't rewrite its own guardrail, settings, or a protected path (holdout data, traffic traces) by any route. It fails closed on unparseable input. For unattended runs, OS-level sandbox configs ship in [`template/sandbox/`](template/sandbox/). -2. **Gates default to REJECT, and their arithmetic is code.** The statistical certification lives in [`tools/verdict.py`](template/tools/verdict.py). A plausible-but-wrong gain can charm an LLM reviewer, but it can't charm an exact sign test that refuses to run without a declared search budget. +| What goes wrong | What stops it here | +|---|---| +| The loop games its own scorecard instead of actually improving | Test data is split so some of it is never used to tune against; hard limits are checked separately from the headline number; metrics are designed so cheating shows up as a worse score, not a better one | +| A result that sounds convincing but isn't real gets approved anyway | `tools/verdict.py` runs an actual statistical significance test. If you don't first say how many things you're going to try, it refuses to run at all, because the more things you try, the more likely one looks good by luck alone | +| The loop edits its own safety rules to hit the number | A gatekeeper script blocks edits to its own configuration and a list of protected files, no matter which tool the loop tries to use | +| The metric itself is broken or noisy, so every check is a coin flip | Before the loop is even allowed to start, a setup check confirms the metric gives a consistent number and can tell a known-good result from a known-bad one | +| The loop runs out of real work and starts busywork to look active | A clean "nothing worked, here's why" counts as a successful run and gets logged as one, not papered over | +| An unattended run gets stuck or runs away with cost | Each run has a time limit and a hard cap on how many steps it can take, so a stuck run gets killed instead of spinning forever | +| The gate itself quietly stops working over time | The gate runs a self-check every time: feed it something meaningless and confirm it still says no | -It comes in **two flavors**, both expressed by the same engine and both shipped as complete worked examples: +Every one of these has a test proving it actually works (219 tests across the two examples), and the automated checks compare the copies in each example against the template so they can't quietly drift apart. -| Flavor | Shape of the problem | Worked example | -|---|---|---| -| **Optimize a running system** | Tune something live: parameters, thresholds, configs. Holdout = out-of-time data. | [`examples/latency-tuner/`](examples/latency-tuner/) | -| **Discover and validate** | Search a candidate space for what genuinely generalizes. Holdout = a held-out split that is never iterated against. | [`examples/prompt-technique-tournament/`](examples/prompt-technique-tournament/) | +Copy `template/`, fill in a handful of blanks, and you have a working loop for your own project. Two complete, working examples ship in the repo so you can see the whole thing running before you build your own. --- -## Why not just prompt "improve yourself in a loop"? +## How it works -Because an unguarded loop fails in predictable ways. Each failure mode here ships with a structural countermeasure, not a polite instruction: +One AI role checks the current state and writes a diagnosis. Another proposes and makes exactly one change. A third role, the **gate**, tries to prove that change is wrong, unsafe, or too good to be true. The change only survives if the gate cannot kill it. This is Anthropic's evaluator-optimizer pattern: one side proposes, the other side is paid to disagree. -| Failure mode | What ships against it | -|---|---| -| The loop **games its own metric** (reward-hacking) | Holdout discipline; hard floors checked with `verdict.py floor`; metrics designed so the hack shows up as a regression (see the latency-tuner's +10s error penalty) | -| A **plausible-but-wrong gain** talks its way past review | `tools/verdict.py`: sign test / paired bootstrap, **Bonferroni-deflated by a candidate budget declared up front**. It fails closed if the budget is undeclared, and its REJECT is binding on the gate agent | -| The loop **edits its own guardrails** | A PreToolUse hook gates Bash + Edit/Write/MultiEdit/NotebookEdit against protected paths and fails closed; the sandbox configs keep the guardrail files read-only at the OS level | -| The metric itself is **broken or noisy**, so every verdict is a coin flip | `tools/metric-contract.sh`, Step 0 of instantiation: the metric must be numeric, deterministic, and score a planted-bad control worse than a known-good one, or the loop doesn't start | -| **Activity theatre**: busywork when nothing real is left | Honest-negative logging ("tested, killed, why" counts as a win); stall detection in the driver; "no change, insufficient data" is a valid iteration | -| A wedged or over-eager iteration **burns the machine or the budget** | Single-flight lock, wall-clock ceiling (`ITER_TIMEOUT`), per-iteration turn budget (`ITER_MAX_TURNS`), resource limits in the sandbox configs | -| The gate itself **rots** | `verdict.py self-test`: an automated known-nothing control, run at the start of every gate review and in CI | +```mermaid +flowchart LR + E["๐Ÿ” Evaluator
looks, doesn't touch"] -->|diagnosis| P["๐Ÿ’ก Proposer
one idea"] + P --> I["๐Ÿ”ง Implementer
makes the change"] + I --> G{"๐Ÿšซ Gate
default: reject it"} + G -->|survives| C["โœ… Keep it, log it"] + G -->|killed| R["โ†ฉ๏ธ Undo it, log it
a clean kill is a win"] + C --> M["๐Ÿ” Meta-improver
can improve the loop itself,
never its own safety rules"] + R --> M +``` + +Two things make this safe to leave running unattended: + +1. **A safety rule the loop physically cannot get around.** Every command and file edit passes through a small script (not a suggestion, an actual gatekeeper) before it runs. That script refuses to let the loop touch its own safety rules or a small list of protected files, and if it can't understand what's being asked, it blocks it rather than guessing. For real unattended use, pair this with an operating-system sandbox; ready-made setups ship in [`template/sandbox/`](template/sandbox/). +2. **The gate's math is code, not opinion.** A change only counts as "proven" if it passes an actual statistical test in [`tools/verdict.py`](template/tools/verdict.py), the same kind of test a scientist would use to check a result wasn't just luck. An AI reviewer can be talked into believing a good story. A statistics script cannot. -Every one of these is exercised by tests (219 across the two examples), and CI diffs the instantiated copies against the template so nothing drifts silently. +There are two shapes this loop commonly takes, and both ship as complete, working examples: + +| Shape | What you're doing | Example | +|---|---|---| +| **Tune something already running** | Nudge live settings (timeouts, thresholds, config values) and check the change still holds up on newer data. | [`examples/latency-tuner/`](examples/latency-tuner/) | +| **Search for the best option** | Try a catalog of candidates and find the one that actually generalizes, not just the one that scored highest on the data you tuned against. | [`examples/prompt-technique-tournament/`](examples/prompt-technique-tournament/) | --- ## The examples -Both examples are **offline, deterministic, and fully instantiated from the template**. Each ships a **planted trap**: a candidate that looks like the best thing on the in-sample data and fails out-of-sample. "The gate catches it" is a tested property here, not a claim. +Both examples run offline with realistic, deterministic fake data (no API key, no network calls), and both are the template fully filled in, so you can read them as a reference. Each one also includes a deliberately planted bad result that looks great on the data you'd normally check and fails on the data you're not supposed to peek at. Proving the gate catches it is part of the test suite, not just a claim in this document. -### ๐Ÿ† `prompt-technique-tournament/`: discover and validate +### ๐Ÿ† `prompt-technique-tournament/`: search for the best option -The loop runs a tournament over prompting techniques (few-shot, chain-of-thought, decomposition, ...) for a support-message triage task, hunting for the technique that generalizes to **unseen cases**, not the one that merely tops the dev leaderboard. +The loop runs a tournament between different prompting techniques (few-shot examples, chain-of-thought, breaking the task into steps, and so on) for a support-message triage task, looking for whichever one actually works on messages it hasn't seen. ```bash cd examples/prompt-technique-tournament @@ -110,26 +128,26 @@ python -m pytest eval/ -q # 129 te claude # then: /iterate ``` -- **The trap:** `keyword_rules`, a surface-token heuristic that wins +2 cases on dev and +0 on holdout. `verdict.py reproduce` rejects it mechanically. -- **The converged champion:** `few_shot+chain_of_thought+decomposition` (35% โ†’ 90%), whose final claim survives an exact sign test deflated by the full 24-candidate budget (11W/0L on holdout, p โ‰ˆ 0.001). -- **Good to know:** three-way split discipline (train = build, dev = iterate, holdout = adjudicate once), and a production seam (`--model claude`) if you want to run it against a real model. Details in [its README](examples/prompt-technique-tournament/README.md). +- **The planted trap:** one technique scores +2 cases better on the tuning data and +0 on the held-back data. The gate catches and rejects it automatically. +- **The winner:** a combination of three techniques takes the pass rate from 35% up to 90%, and that result holds up under a strict statistical check against the full set of candidates that were allowed to compete. +- **Worth knowing:** the data is split three ways (build from one part, tune against a second, check the winner once against a third that's never touched otherwise), and there's an optional switch to run it against a real Claude model instead of the offline fake one. Full details in [its README](examples/prompt-technique-tournament/README.md). -### โšก `latency-tuner/`: optimize a running system +### โšก `latency-tuner/`: tune something already running -The loop tunes a service's retry policy (`timeout_ms`, `retries`, `backoff_ms`) against deterministically replayed traffic. The seed config is breaching its SLO (7.3% errors against a 2% budget). The loop must fix the floor, then cut latency, and every promotion must survive an **out-of-time** holdout window where the upstream has degraded. +The loop tunes a service's retry settings (how long to wait, how many times to retry, how long to back off) against realistic replayed traffic. It has to fix a real problem first (the starting config causes way too many errors), then reduce latency, and every change has to keep working on more recent, worse traffic before it's kept. ```bash cd examples/latency-tuner -python sim/run_eval.py --config config.json --window train # score the champion +python sim/run_eval.py --config config.json --window train # score the current setting python -m pytest sim/ -q # 90 tests, all green claude # then: /iterate ``` -- **The trap:** `timeout_ms=260, retries=3` **tops the train leaderboard** and runs **8.8% errors** (4x the budget) on the holdout window, where congestion persists across retries. `verdict.py floor` kills it. -- **The lesson encoded in the metric:** success-only latency *improves* when you fail fast. That is the classic ops reward-hack, so the headline stat costs every error at +10s and the error floor is checked separately. Both mechanically. -- **Good to know:** time-based holdout (days 6-7 vs 1-5), continuous-metric certification via seeded paired bootstrap, trace checksums pinned in tests so nobody can move the ruler. Details in [its README](examples/latency-tuner/README.md). +- **The planted trap:** a very short timeout with lots of retries looks like the best setting on the older traffic, and causes four times too many errors on the newer, worse traffic. The gate catches and rejects it. +- **The lesson built into the metric:** if you just measure "how fast were the successful requests," failing fast looks like an improvement, because failed requests stop counting. So the real score costs every failure heavily, and a separate hard limit on the error rate is checked no matter what the speed number says. +- **Worth knowing:** the "don't peek at future data" rule here uses time (tune on last week, check against this week) instead of a fixed held-out slice, which is the more common shape for tuning something live. Full details in [its README](examples/latency-tuner/README.md). -**Which one should I read first?** If your problem is "tune the thing that's already running", read the latency-tuner. If it's "find the best candidate out of many", read the tournament. If you're just evaluating Selfsmith, read either README top to bottom; they're short. +**Which one should I read first?** If your problem is "tune the thing that's already running," read the latency-tuner. If it's "find the best option out of several," read the tournament. If you're just getting a feel for Selfsmith, either one works; they're both short. --- @@ -141,136 +159,136 @@ claude # then: /iterate cp -r template/ my-loop/ && cd my-loop ``` -### 2. Prove your metric first (Step 0, don't skip this one) +### 2. Prove your metric works first (don't skip this) -The loop is only as good as the metric it optimizes. Before filling any other blank, open `metric-contract.env` and give it two commands: one that prints the metric for a **known-good** variant, one for a **planted-bad** control. Then: +The loop is only as good as the number it's optimizing. Before you fill in anything else, open `metric-contract.env` and give it two commands: one that prints the metric for something you know is good, one for something you know is bad. Then run: ```bash -bash tools/metric-contract.sh # must print PASS: numeric, deterministic, separating +bash tools/metric-contract.sh # must print PASS ``` -If you can't name a planted-bad control your metric scores worse, you don't have a metric yet, and no downstream gate can save a loop that optimizes a broken measurement. +This checks three things: the metric prints a plain number, it prints the same number twice in a row (so it's not too noisy to trust), and the "good" case actually scores better than the "bad" case. If you can't come up with a bad case your metric correctly scores worse, you don't actually have a working metric yet, and no amount of gate logic downstream can fix that. -### 3. Fill the blanks +### 3. Fill in the blanks -Follow [`INSTANTIATE.md`](template/INSTANTIATE.md), an ordered checklist (GOAL โ†’ PERSONA โ†’ METHODOLOGY โ†’ guardrail โ†’ agents โ†’ iterate command โ†’ run mechanism) with a glossary of every `{{TOKEN}}`. The big ones: +Follow [`INSTANTIATE.md`](template/INSTANTIATE.md), a step-by-step checklist with a glossary of every placeholder. The ones that matter most: -| Token | What it is | +| Placeholder | What goes there | |---|---| -| `{{PROVE_COMMAND}}` | The command that produces your metric. The loop's whole reason to exist | -| `{{VERIFY_COMMAND}}` | Your test/lint gate. Must exit 0 before any change is kept | -| `{{DOMAIN_SAFETY_FLOOR}}` | The constraint no agent may ever trade against the metric | -| `{{DOMAIN_FORBIDDEN_PATTERN}}` / `{{PROTECTED_PATHS}}` | What the guardrail hook hard-blocks (commands / files) | +| `{{PROVE_COMMAND}}` | The command that prints your metric. This is the whole point of the loop. | +| `{{VERIFY_COMMAND}}` | Your normal test or lint command. Must pass before any change is kept. | +| `{{DOMAIN_SAFETY_FLOOR}}` | The one thing that must never be traded away for a better metric. | +| `{{DOMAIN_FORBIDDEN_PATTERN}}` / `{{PROTECTED_PATHS}}` | Commands and files the safety script should block outright. | -While you're in `GOAL.md`, **declare your candidate budget** (how many configs/candidates this campaign may try). `verdict.py` refuses to certify significance without it, by design. +While you're in `GOAL.md`, decide and write down how many different options this campaign is allowed to try. The statistics check refuses to certify a result without that number, on purpose; see the glossary entry on this below. -### 4. Prove the guardrail fires +### 4. Prove the safety script actually blocks things -`bash -n .claude/hooks/guardrail.sh` only proves it parses. Both examples ship a `test_guardrail.py` corpus that feeds the hook real tool-call JSON and asserts the floors return exit 2 on forbidden calls and 0 on benign ones. Copy that pattern and point it at your Floor-1 rules. +Checking that the script has valid syntax (`bash -n .claude/hooks/guardrail.sh`) only proves it doesn't crash. Both examples ship a test file that feeds the script real commands and checks it actually blocks the bad ones and allows the normal ones. Copy that pattern for your own forbidden-command rules. ### 5. Run it -See [Ways to run it](#ways-to-run-it) below. Start manual (`/iterate`), graduate to scheduled once you trust the log. +See [Ways to run it](#ways-to-run-it) below. Start by running it manually, and only move to a schedule once you trust what's in the log. ### What a finished loop looks like -| Part | Role | +| Part | What it does | |---|---| -| `.claude/commands/iterate.md` | **Orchestrator**: a thin router that runs ONE disciplined iteration | -| `.claude/agents/*.md` | **Worker roles**: evaluator, proposer, implementer, gate, meta-improver | -| `.claude/hooks/guardrail.sh` + `settings.json` | **The floor**: a mechanical guardrail the loop can't lift itself | -| `GOAL.md` | Mission, priority order, acceptance test, **candidate budget**, definition of done | -| `PERSONA.md` | Decision philosophy: a named expert archetype and operating principles | -| `METHODOLOGY.md` | "How we don't fool ourselves": the rulebook and the go/no-go gate | -| `LOG.md` | **Memory**: an append-only honest diary, one entry per iteration | -| `tools/verdict.py` + `tools/metric-contract.sh` | **The arithmetic**: mechanical certification plus the prove-your-metric contract | -| `sandbox/` | **The hard boundary**: container / systemd / Seatbelt configs for unattended runs | -| `run-iteration.sh` + `drive-to-goal.sh` | **The drivers**: fresh-session, single-flight, time-boxed, budget-capped ticks | - -### The worker archetypes - -Five roles cover every safe self-improvement loop (simple loops can drop the proposer): - -| Archetype | Tools | Job | +| `.claude/commands/iterate.md` | The orchestrator: routes one run through the steps below, doesn't make changes itself | +| `.claude/agents/*.md` | The five roles: evaluator, proposer, implementer, gate, meta-improver | +| `.claude/hooks/guardrail.sh` + `settings.json` | The safety script the loop cannot edit or route around | +| `GOAL.md` | What the loop is trying to do, in what priority order, and how many options it's allowed to try | +| `PERSONA.md` | A short description of who the loop is "acting as" and how it should reason | +| `METHODOLOGY.md` | The rulebook for what counts as real evidence versus a fluke | +| `LOG.md` | An honest, append-only diary, one entry per run, including the failures | +| `tools/verdict.py` + `tools/metric-contract.sh` | The statistics check and the "does my metric even work" check | +| `sandbox/` | Ready-made setups (container, systemd, macOS) to actually lock the loop down at the operating-system level | +| `run-iteration.sh` + `drive-to-goal.sh` | The scripts that actually run the loop safely on a schedule or back to back | + +### The five roles + +Simple loops can skip the proposer and go straight from evaluator to implementer. + +| Role | Can it write files? | Job | |---|---|---| -| **Evaluator** | read-only | Judge current state vs GOAL; verdict + diagnosis + recommended focus | -| **Proposer** | read-only | ONE mechanism-backed change with a falsifiable expected effect | -| **Implementer** | write | Make the ONE change, test-first, measure, prove it | -| **Gate** | read-only, **VETO** | Adversarially try to kill the change; default-REJECT; `verdict.py` outcomes are binding | -| **Meta-improver** | write | Improve the loop itself; **may never weaken a safeguard** | +| **Evaluator** | No | Reads the current state, writes a diagnosis and a recommendation | +| **Proposer** | No | Suggests exactly one change with a stated reason it should help | +| **Implementer** | Yes | Makes that one change, writes a test for it, measures the result | +| **Gate** | No, but has veto power | Tries to disprove the change; rejects by default when unsure | +| **Meta-improver** | Yes | Improves the loop's own prompts and process, but can never loosen a safety rule | -The gate is the most important archetype. It runs independently, cannot be overridden by the implementer, and must reach a verdict before any change is accepted. +The gate matters most. It runs independently, the implementer can't override it, and nothing gets kept until it says so. --- -## How an iteration works +## How one run works, step by step -One `/iterate` equals one pass through this, in order. Every step that can be a command instead of an opinion, is one. +Each `/iterate` walks through this in order. Anywhere a command can decide the answer instead of an opinion, it does. -1. **Evaluate**: metrics, data health, verdict. A suspected measurement bug outranks every other finding; the metric contract re-runs at triage when in doubt. -2. **Decide ONE change**, from the evaluator's focus and GOAL's priority order. No bundling. -3. **Propose + implement**: a hypothesis with a stated mechanism, then the minimal change, measured. -4. **Gate**: `verdict.py self-test`, then the certification mode that matches the claim (`screen` / `reproduce` / `confirm` / `compare` / `floor`), then the judgment axes (safety, regression, redundancy). REJECT means revert, log, done. -5. **Verify**: the test suite must be green. -6. **Prove**: the metric must actually move as predicted. -7. **Commit**, only after gate + verify + prove. -8. **Log** one honest entry, including kills and no-ops. -9. **Meta-check**: route loop weaknesses to the meta-improver (which may strengthen, but never weaken, a safeguard). -10. **Notify**: signal a human, never wait for one. +1. **Evaluate.** Check the current numbers and data health. A broken measurement always outranks any performance question; if in doubt, the metric check from step 2 above runs again here. +2. **Decide on one change.** Based on the diagnosis and the priorities in `GOAL.md`. Never more than one change at a time. +3. **Propose, then implement.** State a reason the change should help, then make the smallest version of it and measure the result. +4. **Gate it.** Run the self-check, then the matching statistics test, then the human-judgment checks (is this safe, is this a regression, is this redundant). A rejection here is final; the change gets undone. +5. **Verify.** The normal test suite has to pass. A change that breaks tests is never kept, no matter how good the metric looks. +6. **Prove it.** The metric actually has to move the way it was predicted to. +7. **Commit,** only after all three checks above pass. +8. **Log** exactly what happened, wins and honest failures both. +9. **Suggest improvements to the loop itself,** if any came up, routed to the meta-improver (never applied directly by the orchestrator). +10. **Notify** a human that the run finished and what happened. Never wait on a human mid-run; always leave a way to reach one. --- ## Ways to run it -| Mode | Command | When to use it | +| Mode | Command | Best for | |---|---|---| -| **Manual** | `claude` โ†’ `/iterate` | First runs; building trust in the log | -| **Session loop** | `/loop 1h /iterate` | Supervised continuous runs (any interval; omit it and the model self-paces) | -| **Timer** (cron / launchd / systemd) | `0 * * * * cd /path/to/my-loop && ./run-iteration.sh` | **Ongoing** work: tuning a live system where you *want* spacing so new data accumulates | -| **Run until done** | `./drive-to-goal.sh` | **Bounded** work: a finish line to converge on, then stop (DONE marker, stall detection, `MAX_ITERS` cap) | -| **Sandboxed** | `./sandbox/run-sandboxed.sh` | Any unattended run: the same tick, inside the container sandbox | +| **Manual** | `claude` then `/iterate` | Your first few runs, while you're building trust in the log | +| **Session loop** | `/loop 1h /iterate` | Supervised, continuous runs at a fixed interval (or no interval, and the model paces itself) | +| **Scheduled** (cron, launchd, systemd) | `0 * * * * cd /path/to/my-loop && ./run-iteration.sh` | Ongoing tuning, where you want gaps so new data can accumulate | +| **Run until done** | `./drive-to-goal.sh` | A campaign with a finish line: runs back to back until it converges, stalls, or hits a cap | +| **Sandboxed** | `./sandbox/run-sandboxed.sh` | Any unattended run, with an actual operating-system boundary underneath it | -For anything scheduled, always go through **`run-iteration.sh`**, never raw `claude`: +For anything scheduled, always go through `run-iteration.sh`, never call `claude` directly: -- **Fresh session per tick** (`claude -p`, never `--continue`). The loop's memory is on disk (`LOG.md`, champion state, git), so every run re-grounds from truth instead of drifting on stale context. -- **Single-flight lock**: overlapping ticks skip cleanly, with stale-lock reclaim. -- **Time-box**: `ITER_TIMEOUT` (default `50m`) kills a wedged iteration. Needs GNU `timeout`/`gtimeout`; stock macOS has neither until `brew install coreutils`, and the wrapper warns on stderr when running without a ceiling. -- **Spend ceiling**: `ITER_MAX_TURNS` (default `50` agent turns via `--max-turns`). An unattended loop needs a budget limit it cannot argue past, exactly like the safety floor. Raise it deliberately; don't remove it. +- **Every run starts fresh.** No conversation memory carries over; everything the loop needs to know lives on disk (`LOG.md`, git history, the current settings), so it's always working from ground truth, not a stale mental model. +- **Only one run at a time.** If a run is already in progress, a new scheduled tick skips cleanly instead of overlapping it. +- **Each run has a time limit.** `ITER_TIMEOUT` (default 50 minutes) kills a run that's stuck instead of letting it spin forever. +- **Each run has a step limit.** `ITER_MAX_TURNS` (default 50) caps how much work, and cost, a single run can spend. Raise it on purpose if you need to; don't remove it. --- -## The safety model, before you run unattended +## How safe is "unattended"? -Two layers. Know which one you're relying on: +Two separate layers, and it matters which one you're actually relying on: -**Layer 1: the tripwire (ships in `.claude/hooks/`).** A PreToolUse hook gates Bash *and* the write tools against protected paths and forbidden commands, mirrors redirects/`cp`/`tee`/`sed -i` against the write-tool rules, and fails closed on unparseable input. It catches honest mistakes cold. But a string-matching deny-list is **not a security boundary**: shell quoting and obfuscation can evade any regex. +**Layer 1: the gatekeeper script.** Every command and file edit is checked against a list of rules before it runs, and if the check can't make sense of what's being asked, it blocks by default rather than guessing. This catches honest mistakes reliably. What it is not: a real security boundary against a determined attempt to get around it. Clever quoting or encoding can slip past any list of text patterns. -**Layer 2: the OS sandbox (ships in [`template/sandbox/`](template/sandbox/)).** OS-level enforcement where destructive syscalls are impossible regardless of what the loop tries: +**Layer 2: an actual operating-system sandbox.** This is the real boundary, where the operating system itself refuses to allow certain actions no matter what the loop tries. Three ready-made setups ship in [`template/sandbox/`](template/sandbox/): -| Option | Strength | Files | +| Option | How strong | What's there | |---|---|---| -| Container (any OS) | Strongest, portable | `Dockerfile`, `run-sandboxed.sh`: read-only rootfs, dropped capabilities, pid/memory ceilings, guardrail files mounted read-only *inside* the writable area | -| systemd hardened unit (Linux) | Strong, no container runtime | `selfsmith-loop.service` + `.timer`: `ProtectSystem=strict`, `ReadOnlyPaths` on the guardrail files, syscall filter | -| Seatbelt profile (macOS) | Good, built-in | `claude-loop.sb`: deny all writes except the working set, guardrail files re-denied inside it | +| Container | Strongest, works anywhere | A read-only filesystem, no elevated permissions, memory/CPU caps, and the safety files locked read-only even inside the one folder the loop can write to | +| systemd (Linux) | Strong, no container needed | A hardened service definition with the same restrictions built into the operating system directly | +| macOS Seatbelt | Good, built into macOS | Blocks all writes except the loop's own folder, and blocks the safety files even there | -Also distinguish the **mechanical floor** (hook + sandbox, hard blocks) from the **disciplinary controls** (gate veto, meta-improver restraint, LLM judgment). The judgment layers make the loop *productive*; only the mechanical layers make it *safe*. And verify the sandbox once before trusting it: from inside, `touch .claude/hooks/x` and `touch /etc/x` must both fail with permission errors. A sandbox you haven't seen refuse a write is a hypothesis. +The gatekeeper script (layer 1) and the sandbox (layer 2) are hard technical limits. The gate's veto and the loop's own good judgment are not; they're the AI reasoning well, which is valuable but not a guarantee. Before trusting a sandbox, test it: from inside, try to edit the safety files and try to write outside the loop's own folder. Both should fail with a permission error. If you haven't seen it refuse, you don't actually know it works yet. --- -## The 10 principles +## The 10 rules -These are the core of the design. Carry them verbatim when instantiating. +The short version of everything above, useful as a checklist once you understand the reasoning. Keep these as-is when you build your own loop. -1. Doing and checking are **never the same role**: independent adversarial review with veto. -2. **One change per iteration**: reversible, logged, attributable. -3. The loop **can't lift its own floor**: hook (mechanical, gates Bash + write tools, fails closed) + gate (judgment) + meta-improver barred from loosening safeguards. The deny-list is a tripwire; the real boundary for unattended runs is an OS sandbox. -4. The orchestrator is a **thin router**: triage โ†’ route โ†’ decide, never implement. -5. Gates **default to REJECT**: killing a good idea is cheaper than shipping a bad one. -6. **Honest negatives count**: "tested, killed, why" is a win. No activity theatre. -7. **Never block on a human, always able to reach one** (notify escape hatch). -8. **Priority order puts correctness/safety above the headline metric**: never optimize on a broken measurement. -9. **Meta-improvement is first-class but separate**: the loop gets better at getting better. -10. **Reward-hacking is an explicit threat**: guardrails against the loop gaming its own metrics. +1. The role that makes a change and the role that checks it are never the same one. +2. Never more than one change per run. Reversible, logged, and easy to trace back to a cause. +3. The loop cannot loosen its own safety rules: the gatekeeper script blocks it mechanically, the gate blocks it by judgment, and the meta-improver is explicitly barred from doing it. The script alone is not a full security boundary; pair it with an operating-system sandbox for real unattended use. +4. The orchestrator only routes and decides. It never makes changes itself. +5. When in doubt, the gate says no. A missed improvement costs less than a shipped mistake. +6. An honest "nothing worked, here's why" is a successful run, not a failure to hide. +7. Never wait on a human mid-run, but always leave a way to reach one. +8. Correctness and safety always outrank the headline number. Never optimize against a measurement you suspect is broken. +9. The loop can improve its own process, but that's a separate, explicitly labeled kind of change from improving the thing it's working on. +10. Assume the loop will try to game its metric if given the chance, and build the countermeasure in up front rather than reacting after the fact. --- @@ -279,69 +297,86 @@ These are the core of the design. Carry them verbatim when instantiating. ``` selfsmith/ โ”œโ”€โ”€ template/ โ† copy this to start your own loop -โ”‚ โ”œโ”€โ”€ INSTANTIATE.md โ† the ordered fill-in checklist (start here) -โ”‚ โ”œโ”€โ”€ metric-contract.env โ† Step 0: prove your metric first +โ”‚ โ”œโ”€โ”€ INSTANTIATE.md โ† the step-by-step checklist, start here +โ”‚ โ”œโ”€โ”€ metric-contract.env โ† step 0: prove your metric works โ”‚ โ”œโ”€โ”€ GOAL.md ยท PERSONA.md ยท METHODOLOGY.md ยท LOG.md โ”‚ โ”œโ”€โ”€ .claude/ -โ”‚ โ”‚ โ”œโ”€โ”€ commands/iterate.md โ† the orchestrator slash command -โ”‚ โ”‚ โ”œโ”€โ”€ agents/ โ† the five worker archetypes -โ”‚ โ”‚ โ”œโ”€โ”€ hooks/guardrail.sh โ† the mechanical floor (fails closed) -โ”‚ โ”‚ โ””โ”€โ”€ settings.json โ† wires the hook to Bash + write tools +โ”‚ โ”‚ โ”œโ”€โ”€ commands/iterate.md โ† the orchestrator +โ”‚ โ”‚ โ”œโ”€โ”€ agents/ โ† the five roles +โ”‚ โ”‚ โ”œโ”€โ”€ hooks/guardrail.sh โ† the safety script the loop cannot edit +โ”‚ โ”‚ โ””โ”€โ”€ settings.json โ† wires the safety script into every command โ”‚ โ”œโ”€โ”€ tools/ -โ”‚ โ”‚ โ”œโ”€โ”€ verdict.py โ† mechanical gate certification (stdlib only) -โ”‚ โ”‚ โ””โ”€โ”€ metric-contract.sh โ† the prove-your-metric contract -โ”‚ โ”œโ”€โ”€ sandbox/ โ† Docker, systemd, and Seatbelt configs + README -โ”‚ โ”œโ”€โ”€ run-iteration.sh โ† fresh-session, locked, time-boxed, budget-capped tick -โ”‚ โ””โ”€โ”€ drive-to-goal.sh โ† run ticks back-to-back until convergence +โ”‚ โ”‚ โ”œโ”€โ”€ verdict.py โ† the statistics check (no external dependencies) +โ”‚ โ”‚ โ””โ”€โ”€ metric-contract.sh โ† the "does my metric actually work" check +โ”‚ โ”œโ”€โ”€ sandbox/ โ† container, systemd, and macOS lockdown setups +โ”‚ โ”œโ”€โ”€ run-iteration.sh โ† runs one safe, timed, single-flight tick +โ”‚ โ””โ”€โ”€ drive-to-goal.sh โ† runs ticks back to back until done โ””โ”€โ”€ examples/ - โ”œโ”€โ”€ prompt-technique-tournament/ โ† discover-and-validate flavor, fully instantiated - โ””โ”€โ”€ latency-tuner/ โ† optimize-a-running-system flavor, fully instantiated + โ”œโ”€โ”€ prompt-technique-tournament/ โ† "search for the best option," fully filled in + โ””โ”€โ”€ latency-tuner/ โ† "tune something already running," fully filled in ``` -CI runs both example suites and enforces **parity**: the runners, `verdict.py`, `metric-contract.sh`, and the guardrail's machine-safety floor are byte-diffed between the template and every example, so instantiated copies cannot silently drift. +Every push runs both examples' test suites, and a separate check confirms the shared scripts in each example still match the template exactly, so a copy can't silently drift out of sync. + +--- + +## Glossary + +Terms used above, defined once here so the rest of the document doesn't have to keep re-explaining them. + +| Term | Means | +|---|---| +| **Gate** | The role that tries to disprove a proposed change before it's kept. Says no by default. | +| **Fails closed** | If a safety check can't understand or verify something, it blocks the action instead of allowing it through. The opposite, failing open, is what you don't want. | +| **Holdout** | Data that's set aside and never used to tune against. It only ever gets checked once, to confirm a result that already looked good is real and not a fluke. | +| **Candidate budget** | How many different options you've committed to trying in a campaign, decided before you start. The more options you try, the more likely one looks good purely by chance, so the statistics check needs to know this number to correct for it. | +| **Statistical significance test** | A calculation that answers "how likely is it this result happened by chance instead of because the change actually worked?" `verdict.py` runs an actual one of these instead of asking an AI to eyeball the numbers. | +| **Reward-hacking** | When something being optimized finds a way to score better without actually getting better at the real task. | +| **Guardrail / floor** | A hard rule the loop is never allowed to cross, checked mechanically rather than left to judgment. | +| **Sandbox** | An operating-system-level lockdown where certain actions are physically impossible, not just discouraged. | --- ## FAQ **Do I need an API key?** -Not for the examples; they're offline and deterministic (the tournament has an optional `--model claude` seam for real calls). Your own loop runs inside your normal Claude Code session/subscription; scheduled headless runs (`claude -p`) authenticate however your CLI already does. +Not for the examples; they run offline with realistic fake data (the tournament example has an optional switch to use a real Claude model instead). Your own loop runs inside your normal Claude Code session or subscription; a scheduled headless run authenticates however your Claude Code CLI already does. -**What does an iteration cost?** -One `/iterate` is a handful of subagent calls (the gate defaults to a stronger model; see the `model:` frontmatter in `agents/*.md` and change it if you want cheaper runs). Unattended ticks are capped by `ITER_MAX_TURNS` (default 50 turns) and `ITER_TIMEOUT` (default 50m), so a runaway iteration is cut off mechanically. +**What does one run cost?** +A handful of subagent calls. The gate defaults to a stronger model (see the `model:` line in each file under `agents/`), which you can change if you want cheaper runs. Scheduled runs are capped by `ITER_MAX_TURNS` and `ITER_TIMEOUT`, so a run that goes wrong stops instead of spending indefinitely. **My metric isn't a test score. Can I still use this?** -If a shell command can print it as a number, yes. That's the whole interface (`{{PROVE_COMMAND}}`, plus the Step 0 contract). If no command can print it, build that first; the loop cannot optimize what it cannot measure. +If a command can print it as a plain number, yes; that's the entire requirement. If nothing can print it yet, build that first. The loop can't improve something it can't measure. -**What if my domain has no natural holdout?** -Use time: tune on last week, adjudicate on this week (the latency-tuner shows exactly this). The only rule is that the adjudication data must never be iterated against. +**What if I don't have an obvious way to split my data into a holdout?** +Use time instead: tune against last week's data, check the result against this week's. The latency-tuner example does exactly this. The one rule either way is that the checking data never gets used to make tuning decisions. -**Why does `verdict.py` refuse to run without `--search-size`?** -Because "we tried N things and this one won" needs deflating by N, including the candidates you tried and quietly dropped. Undeclared search is the main way loops fool their authors, so the tool fails closed rather than defaulting to 1. Declare your budget in `GOAL.md` up front. +**Why does the statistics check refuse to run without a stated candidate budget?** +Because "I tried a few things and this one won" is meaningless without knowing how many things you tried. The more options you test, the more likely one looks good by luck alone, and the check corrects for that. Rather than guess or default to a low number, it simply refuses to certify anything until you tell it the real number. **Can the loop edit its own prompts?** -Yes. That's the meta-improver, and it's first-class. What it can never do is weaken a safeguard: those changes must go to the gate and page a human, and the guardrail files themselves are hook-protected and (in the sandbox) OS-read-only. +Yes, that's the meta-improver's job, and it's a normal, expected part of the loop. What it can never do is loosen a safety rule on its own authority. Any change like that has to go through the gate and notify a human. -**How do I trust it enough to leave it running?** -Read `LOG.md` after a few supervised iterations. A trustworthy loop's diary contains kills, no-ops, and reverts, not just wins. Then run it in the sandbox anyway. +**How do I know I can trust it enough to leave it running?** +Read the log after a few supervised runs. A trustworthy log has real failures in it, changes that got tried and undone, not just wins. Then still run it inside a sandbox. --- ## Contributing -Contributions are very welcome. This template exists to be useful to the community, and the best way to improve it is more real-world instantiations. +Contributions are welcome. This template is meant to be genuinely useful outside of its original use case, and the best way to improve it is more real examples built from it. -Especially valuable: +Especially useful: -- **New worked examples**: a third domain (code performance, RAG retrieval quality, CI flakiness, agent prompt tuning, ...) instantiated from the template. Follow the two existing examples' shape: offline/deterministic if possible, a planted trap the gate must catch, a full test corpus, and the shared files (`run-iteration.sh`, `drive-to-goal.sh`, `tools/`) byte-identical to the template. CI enforces this. -- **Guardrail bypass reports**: found a Bash construction that slips Floor 2, or a payload the parser mishandles? Open an issue or PR with the case added to a `test_guardrail.py` corpus, pinned closed. -- **Sandbox configs**: hardened profiles for platforms not covered yet (Windows, BSD, other container runtimes). -- **Methodology pitfalls**: a failure mode you hit running a loop of your own, written up as a `METHODOLOGY.md` entry with the countermeasure that fixed it. +- **A third worked example** in a new domain (code performance, search quality, flaky tests, anything measurable). Follow the shape of the two existing examples: offline and repeatable if possible, a planted bad result the gate has to catch, a full test suite, and the shared scripts kept identical to the template (this is checked automatically). +- **Bug reports against the safety script**: found a command that should have been blocked and wasn't? Open an issue or a PR with the exact case added to the test file, so it stays blocked for good. +- **More sandbox setups** for platforms not covered yet (Windows, BSD, other container tools). +- **Real lessons learned**, written up as an addition to a `METHODOLOGY.md` with the fix that addressed it, from running a loop of your own. -Ground rules (the same rules the loop itself lives by): keep template/example parity green, never weaken a shipped safeguard or the fail-closed behavior, and every claimed protection needs a test that proves it fires: a positive control the check must catch, plus a clean negative. +The same rules the loop lives by apply here: keep the template and examples in sync, never weaken a safety check or its fail-closed behavior, and any claimed protection needs a test proving it actually fires, including a case that should pass and a case that should fail. ```bash -# the full local check before you open a PR +# run this before opening a PR (cd examples/prompt-technique-tournament && python -m pytest eval/ -q) (cd examples/latency-tuner && python -m pytest sim/ -q) python3 template/tools/verdict.py self-test diff --git a/examples/latency-tuner/README.md b/examples/latency-tuner/README.md index ca05063..bd6d7b3 100644 --- a/examples/latency-tuner/README.md +++ b/examples/latency-tuner/README.md @@ -1,14 +1,8 @@ -# Latency tuner: example loop (optimize-a-running-system flavor) +# Latency tuner: example loop (tune something already running) -A self-improving loop that tunes a running service's retry policy (`config.json`: -`timeout_ms`, `retries`, `backoff_ms`) against replayed production traffic, and only -promotes changes that survive an **out-of-time** holdout window. Fully offline and -deterministic: no API key, no network. +A self-improving loop that tunes a service's retry settings (`config.json`: how long to wait, how many times to retry, how long to back off between attempts) against realistic replayed traffic, and only keeps a change if it still holds up on newer, worse traffic. Fully offline: no API key, no network calls. -This is the second flavor described in the root README: where the -[prompt-technique tournament](../prompt-technique-tournament/) *discovers and validates* a -winner from a candidate catalog, this loop *optimizes a running system*: incremental -config changes, a time-based holdout, and a hard operational floor. +New to Selfsmith? The [root README](../../README.md) explains the whole idea in plain terms and has a glossary for any term below that isn't obvious. This example is the "tune something already running" shape described there; the [prompt-technique tournament](../prompt-technique-tournament/) is the "search for the best option" shape. ## Prerequisites @@ -17,45 +11,37 @@ config changes, a time-based holdout, and a hard operational floor. ## What this loop does -The seed policy is breaching its SLO (7.3% errors against a 2% budget). Each iteration, -the loop (`/iterate`): +The starting settings have a real problem: 7.3% of requests fail, against a 2% budget. Each run (`/iterate`): -1. Evaluates the champion config on the train window (days 1-5). -2. Proposes ONE knob change with a stated mechanism, and the traffic regime it assumes. -3. Measures it on train (mean **effective** latency: errors costed at +10,000ms). -4. Gates it mechanically: `tools/verdict.py compare` (paired bootstrap, deflated by the - declared 24-candidate budget), the 2% error floor on BOTH windows, and holdout - non-inferiority (days 6-7, where the upstream has degraded and congestion persists). -5. Logs the result and updates `config.json` only on approval. +1. Checks how the current settings are doing against the older, healthier traffic. +2. Proposes changing exactly one setting, with a reason it should help. +3. Scores that change against the older traffic (the score is "effective latency": how long a request took, with every failure counted as if it took 10 seconds). +4. Checks the change three ways: does it actually help by more than could be chance, does the error rate stay under budget on both the older and newer traffic, and does it still hold up on the newer, worse traffic. +5. Writes down the result and updates `config.json`, but only if the change passed every check. -**The planted trap:** `timeout_ms=260, retries=3` tops the train leaderboard (mean -effective โ‰ˆ 320ms vs the honest optimum โ‰ˆ 332ms) because train slowness is transient and -retries escape it. On the holdout window congestion persists across attempts, so the same -config runs **8.8% errors** (4ร— the floor) and mean effective 1211ms. The gate must kill -the best-looking config on the board. The converged champion is -`timeout 2000 / retries 2 / backoff 0` (train 332ms, holdout 299ms, 0% errors both). +**The planted trap:** a very short timeout with three retries looks like the best setting on the older traffic (average score about 320ms versus 332ms for the honest winner) because on that traffic, slowdowns are brief and a quick retry usually escapes them. On the newer traffic, slowdowns last longer and a retry usually lands in the middle of the same slowdown instead of escaping it, so that same setting causes 8.8% failures, more than four times the budget. The gate has to catch and reject the best-looking setting on the board, and the test suite proves it does. The setting that actually wins is 2000ms timeout / 2 retries / no backoff delay, scoring 332ms on the older traffic and 299ms on the newer traffic, with zero failures on both. ## Run the harness directly ```bash cd examples/latency-tuner -# Score the current champion on the train window +# Score the current setting against the older traffic python sim/run_eval.py --config config.json --window train -# Score a candidate without touching config.json +# Try a different setting without touching config.json python sim/run_eval.py --window train --override timeout_ms=260 --override retries=3 --override backoff_ms=20 -# Adjudicate on holdout (once per challenger, after a train win; never to explore) +# Check a candidate against the newer traffic (only do this once, to confirm a winner) python sim/run_eval.py --window holdout --override timeout_ms=260 --override retries=3 --override backoff_ms=20 -# Dump paired per-request vectors and certify mechanically +# Save the per-request numbers and run the actual statistics check python sim/run_eval.py --window train --dump /tmp/champ.txt python sim/run_eval.py --window train --override timeout_ms=260 --override retries=3 --dump /tmp/trap.txt python tools/verdict.py compare --champion /tmp/champ.txt --challenger /tmp/trap.txt \ --search-size 24 --stat mean --direction lower -# Run all harness tests +# Run everything python -m pytest sim/ -q ``` @@ -64,35 +50,35 @@ python -m pytest sim/ -q ```bash claude # open Claude Code in this directory # inside the session: -/iterate # ONE iteration, manually -/loop 1h /iterate # CONTINUOUS, hourly (a timer suits ongoing tuning) +/iterate # ONE run, manually +/loop 1h /iterate # CONTINUOUS, hourly (a schedule suits this kind of ongoing tuning) ``` -## The discipline at a glance +## The rules this example follows, at a glance -| Mechanism | Where | +| Rule | How it's enforced | |---|---| -| Fixed measurement data | `traces/*.jsonl` are protected paths; `make_traces.py` blocked in-loop (guardrail Floor 1) | -| Reward-hack countermeasure | effective latency (+10s per error) as headline stat AND a mechanical 2% error floor | -| Multiple-comparisons discipline | `verdict.py compare --search-size 24` (budget declared in GOAL.md, fixed up front) | -| Out-of-time validation | holdout = later window, degraded upstream, persistence 0.75 vs train 0.30 | -| Metric contract | `bash tools/metric-contract.sh` (numeric, deterministic, separating) | +| The traffic data itself never changes | `traces/*.jsonl` is a protected path; the script that generated it is blocked while the loop is running | +| Failing fast can't be disguised as an improvement | The score counts every failure as a heavy time penalty, and the failure rate is checked separately no matter what the speed number says | +| Trying lots of settings doesn't let a lucky one sneak through | The statistics check is told upfront how many settings (24) this campaign is allowed to try, and corrects for that | +| A setting that only works on the past doesn't get promoted | The newer, worse traffic (days 6-7 versus days 1-5) has to confirm the result before it's kept | +| The metric itself is trustworthy | `bash tools/metric-contract.sh` confirms it's a plain number, consistent run to run, and can tell good settings from bad ones | ## Files at a glance ``` .claude/ - commands/iterate.md # orchestrator (Rae) - agents/ # evaluator, proposer, implementer, gate, meta-improver - hooks/ # guardrail: blocks trace regeneration/mutation + commands/iterate.md # the orchestrator (goes by "Rae" in this example) + agents/ # evaluator, proposer, implementer, gate, meta-improver + hooks/ # the safety script: blocks touching the traffic data sim/ - run_eval.py # deterministic replay scorer (--window, --override, --dump) - make_traces.py # trace provenance; run ONCE at project start, blocked in-loop - test_*.py # harness, guardrail, verdict, and metric-contract corpora -traces/ # train.jsonl (days 1-5), holdout.jsonl (days 6-7): the ruler -tools/ # verdict.py (mechanical gate), metric-contract.sh -config.json # the SEED policy (2000/0/0, SLO-breaching); the loop walks it forward -GOAL.md # mission, floors, candidate budget, done definition -METHODOLOGY.md # window discipline, the physics, go/no-go gate -LOG.md # worked history: promote, trap rejection, promote + run_eval.py # scores a setting against replayed traffic (--window, --override, --dump) + make_traces.py # generated the traffic data once, at the start; blocked while the loop runs + test_*.py # the full test suite covering the harness, the safety script, and the statistics check +traces/ # train.jsonl (days 1-5), holdout.jsonl (days 6-7): the fixed traffic data +tools/ # verdict.py (the statistics check), metric-contract.sh +config.json # the starting settings (2000ms/0 retries/0 backoff, too many failures); the loop walks this forward +GOAL.md # the mission, the hard limits, how many settings this campaign may try, what "done" means +METHODOLOGY.md # the data-split rules, why this domain is easy to fool yourself in, and what it takes to promote a winner +LOG.md # a worked example history: one win, one caught trap, one more win ``` diff --git a/examples/prompt-technique-tournament/README.md b/examples/prompt-technique-tournament/README.md index 413efc6..54205ed 100644 --- a/examples/prompt-technique-tournament/README.md +++ b/examples/prompt-technique-tournament/README.md @@ -1,6 +1,8 @@ # Prompt-technique tournament: example loop -A self-improving loop that runs a tournament over prompting techniques for a fixed use-case (support-message triage) and converges on the technique that generalizes best to **unseen cases**. The harness is offline and deterministic by default; no API key needed. +A self-improving loop that tests different prompting techniques against a fixed task (sorting support messages into categories) and finds whichever one actually works on messages it hasn't seen before, not just the one that scored best on the messages it practiced on. Runs offline with realistic fake data; no API key needed. + +New to Selfsmith? The [root README](../../README.md) explains the whole idea in plain terms and has a glossary for any term below that isn't obvious. ## Prerequisites @@ -10,15 +12,15 @@ A self-improving loop that runs a tournament over prompting techniques for a fix ## What this loop does -Each iteration, the loop (`/iterate`): +Each run (`/iterate`): -1. Evaluates the current champion's dev and holdout scores. -2. Proposes one technique change (or combo). -3. Measures it on dev. -4. Gates it: promotes only if the dev gain reproduces on the **never-tuned holdout** and clears the noise floor. -5. Logs the result (win or rejection) and updates the champion if approved. +1. Checks how the current best technique is doing. +2. Proposes one technique to try, or one combination of techniques. +3. Scores it against the tuning data. +4. Decides: the new technique only wins if it also holds up on data that was never used for tuning, and the improvement is big enough to trust. +5. Writes down what happened, win or loss, and updates the current best technique if it won. -The planted trap, `keyword_rules`, gains +2 cases on dev but +0 on holdout. The gate must catch it. The converged champion is `few_shot+chain_of_thought+decomposition` at 90% dev / 90% holdout. +There's a deliberately planted bad result in the catalog: a technique called `keyword_rules` scores +2 cases better on the tuning data and +0 better on the data set aside for checking. The gate has to catch that and reject it, and the test suite proves it does. The technique that actually wins is a combination called `few_shot+chain_of_thought+decomposition`, scoring 90% on both the tuning data and the checking data. ## Run one iteration @@ -30,70 +32,70 @@ claude # opens Claude Code in this directory Inside Claude Code: ``` -/iterate # ONE iteration, manually -/loop 30m /iterate # CONTINUOUS: one iteration every 30 min (pick any interval) +/iterate # ONE run, manually +/loop 30m /iterate # CONTINUOUS: one run every 30 minutes (pick any interval) ``` The loop is offline by default: no API key, no paid calls, safe to run unattended. ## Run the harness directly -Score any technique on any split: +Score any technique against any part of the data: ```bash -# Score the champion combo on dev +# Score the winning combination against the tuning data python eval/run_eval.py --technique few_shot+chain_of_thought+decomposition --split dev -# Score it on holdout (do this only to adjudicate, not to iterate) +# Score it against the held-back checking data (only do this to confirm a winner, not to explore) python eval/run_eval.py --technique few_shot+chain_of_thought+decomposition --split holdout -# Test the planted trap +# Try the planted bad result python eval/run_eval.py --technique keyword_rules --split dev python eval/run_eval.py --technique keyword_rules --split holdout ``` -Run all harness tests: +Run everything: ```bash python -m pytest eval/ -q # expects all green ``` -## The split discipline +## The three-way data split -| Split | Use | +| Split | Used for | |---|---| -| `train` | Inspect cases, build few-shot example pools. | -| `dev` | Measure every candidate technique. Working signal. | -| `holdout` | Adjudicate a challenger only, after it wins on dev. **Never inspect for technique ideas.** | +| `train` | Building examples, looking at cases, getting a feel for the task. | +| `dev` | Scoring every candidate technique. This is the working signal. | +| `holdout` | Confirming a winner, once, after it already won on `dev`. Never used to come up with ideas. | -The holdout is the one honest judge. Iterating against it, even by looking at its cases for inspiration, invalidates the result. The guardrail in `.claude/settings.json` hard-blocks any command that would mutate `holdout.jsonl` inside the loop. +`holdout` is the one honest check. Looking at it for inspiration, even without formally scoring against it, defeats the whole point, the same way peeking at exam answers while studying does. The safety script in `.claude/settings.json` blocks any command that would edit `holdout.jsonl` while the loop is running, so this isn't just a rule someone has to remember. -## Production seam +## Running against a real model instead of the offline fake one -By default the harness uses an offline mock model (deterministic, keyed off each case's `_solved_by` metadata; see `eval/task.md`). To run against a real Anthropic model: +By default the harness uses a fake, deterministic stand-in for a model (its answers are decided by metadata attached to each test case; see `eval/task.md`). To run it against an actual Claude model instead: ```bash export ANTHROPIC_API_KEY=sk-ant-... python eval/run_eval.py --technique few_shot+chain_of_thought --split dev --model claude ``` -`--model claude` calls `claude-haiku-4-5-20251001` via the Anthropic SDK. See the `claude-api` skill for current model IDs, pricing, and SDK usage before changing the model. The loop itself stays on `--model mock` by the guardrail in `.claude/hooks/`; `--model claude` is for one-off human validation only. +`--model claude` calls `claude-haiku-4-5-20251001` through the Anthropic SDK. Check the `claude-api` skill for current model names and pricing before changing which model it uses. The loop itself is locked to the offline fake model by the safety script in `.claude/hooks/`; `--model claude` is meant for a human to check by hand, not for the loop to use on its own. ## Files at a glance ``` .claude/ - commands/iterate.md - orchestrator (Vera) - agents/ - evaluator, proposer, implementer, gate, meta-improver - hooks/ - guardrail: blocks holdout mutation and paid model calls + commands/iterate.md - the orchestrator (goes by "Vera" in this example) + agents/ - evaluator, proposer, implementer, gate, meta-improver + hooks/ - the safety script: blocks editing holdout data or calling a paid model eval/ - run_eval.py - scorer (--technique, --split, --model) - techniques.py - full technique catalog + renderers - cases/ - train.jsonl, dev.jsonl, holdout.jsonl - champion.json - loop's SEED/start state (zero_shot, 35%); the loop walks it forward. The converged champion (few_shot+chain_of_thought+decomposition, 90%) is shown in eval/leaderboard.md - leaderboard.md - full ranked history including rejections -GOAL.md - what "done" looks like -PERSONA.md - Vera's principles -METHODOLOGY.md - split discipline + go/no-go gate -LOG.md - append-only iteration history + run_eval.py - scores a technique (--technique, --split, --model) + techniques.py - the full catalog of techniques and how each one changes the prompt + cases/ - train.jsonl, dev.jsonl, holdout.jsonl + champion.json - the starting point (zero_shot, 35%); the loop walks this forward over time. The eventual winner (few_shot+chain_of_thought+decomposition, 90%) is recorded in eval/leaderboard.md + leaderboard.md - the full history, including the techniques that lost +GOAL.md - what "done" means for this loop +PERSONA.md - Vera's guiding principles +METHODOLOGY.md - the data-split rules and what it takes to promote a winner +LOG.md - an append-only diary, one entry per run ```