Harden engine error handling; 4x tests (24->136) and demos (5->20)

README.md

@@ -14,7 +14,7 @@ Ask yourself:
 
 A lot of useful work has that one shape: produce a candidate, check it against a goal, use the feedback to produce a better one — until it's good enough, it stops improving, or you run out of budget. Solvers, optimizers, retry-with-feedback agent loops, and self-correcting generators are all *this loop*.
 
-`cyclework` makes it a first-class, inspectable object instead of a `while` buried in a function. You supply two functions — how to **check** a candidate and how to **revise** it given feedback — and the engine runs the cycle, detects convergence and plateaus, enforces a budget, captures errors, and hands back a full trace.
+`cyclework` makes it a first-class, inspectable object instead of a `while` buried in a function. You supply two functions — how to **check** a candidate and how to **revise** it given feedback — and the engine runs the cycle, detects convergence and plateaus, enforces a budget, captures errors (including in the optional `seed`), validates its own configuration, and hands back a full trace.
 
 - **Domain-agnostic.** The candidate ("state") can be a number, a string, a dict, an AST — anything.
 - **Inspectable.** Every iteration is recorded; you get the answer *and* the path to it.
@@ -108,7 +108,7 @@ model, stopping conditions, and why the loop is a first-class object).
 - **Solved** — a verdict comes back `ok=True`.
 - **Plateau** — with `patience=N` set, the score fails to improve by at least `min_delta` for `N` consecutive checks. (Higher-is-better; negate a cost to minimize it.)
 - **Exhausted** — `max_iterations` reached without solving.
-- **Error** — a `check`/`revise` raised; the run aborts cleanly with the trace so far.
+- **Error** — a `check`, `revise`, or `seed` raised (or `check` returned something other than a `Verdict`); the run aborts cleanly with a message naming the stage and iteration, and the trace + best candidate captured so far.
 
 ### Observability
 
@@ -120,12 +120,12 @@ Engine(check, revise, on_iteration=lambda it: print(it.index, it.score))
 
 ## Demos
 
-Five runnable, narrated scenarios in [`demos/`](demos/), each for a different
-audience — all driving the real API, no network, exit 0. Full descriptions in
-[`docs/DEMOS.md`](docs/DEMOS.md).
+Twenty runnable, narrated scenarios in [`demos/`](demos/), each for a different
+audience or facet of the engine — all driving the real API, no network, exit 0.
+Full descriptions in [`docs/DEMOS.md`](docs/DEMOS.md).
 
 ```bash
-python demos/run_all.py               # all five, end to end
+python demos/run_all.py               # all twenty, end to end
 python demos/02_numeric_solvers.py    # or just one
 ```
 
@@ -136,14 +136,15 @@ python demos/02_numeric_solvers.py    # or just one
 | 3 | [`03_plateau_and_budget.py`](demos/03_plateau_and_budget.py) | Production loop owners | Honest stopping: SOLVED / PLATEAU / EXHAUSTED / ERROR, always keeping the best candidate. |
 | 4 | [`04_feedback_refiner.py`](demos/04_feedback_refiner.py) | Pipeline engineers | Feedback-driven revision: the verdict names the fix, revise applies exactly it. |
 | 5 | [`05_streaming_observability.py`](demos/05_streaming_observability.py) | Observability engineers | Watching a loop live via `on_iteration` and the `Engine.cycle()` generator. |
+| 6–20 | [see `docs/DEMOS.md`](docs/DEMOS.md) | resilience, ML, data-cleaning, compliance, reliability, numerical … | Error recovery, gradient descent, constraint repair, retry-with-backoff, audit logging, best-of non-convergent search, Newton vs bisection, and more. |
 
 On a cp1252 (Windows) console, prefix with `PYTHONUTF8=1`.
 
 ## Testing
 
 ```bash
 pip install -e ".[dev]"
-pytest -q          # 24 tests (engine, examples, and demo smoke tests)
+pytest -q          # 136 tests (engine, data model, examples, and demo smoke tests)
 ```
 
 ## License

cyclework/engine.py

@@ -11,8 +11,11 @@
 
 stopping when a verdict is `ok`, when the score plateaus (no improvement of at
 least `min_delta` for `patience` checks), or when `max_iterations` is reached.
-A stage raising an exception aborts cleanly with `Status.ERROR` and the trace
-collected so far.
+Any stage (`seed`, `check`, `revise`) that raises — or a `check` that returns
+something other than a `Verdict` — aborts cleanly with `Status.ERROR`, a message
+naming the stage and iteration, and the trace collected so far. Constructor
+arguments are validated up front (callable stages; non-negative budget/min_delta;
+patience >= 1).
 
 Convention: `score` is higher-is-better. For a "lower is better" objective
 (an error/cost), return its negation as the score.
@@ -47,6 +50,16 @@ def __init__(
             raise ValueError("max_iterations must be >= 0")
         if patience is not None and patience < 1:
             raise ValueError("patience must be >= 1")
+        if min_delta < 0:
+            raise ValueError("min_delta must be >= 0")
+        if not callable(check):
+            raise TypeError("check must be callable: check(state) -> Verdict")
+        if not callable(revise):
+            raise TypeError("revise must be callable: revise(state, verdict) -> state")
+        if seed is not None and not callable(seed):
+            raise TypeError("seed must be callable: seed(initial) -> state")
+        if on_iteration is not None and not callable(on_iteration):
+            raise TypeError("on_iteration must be callable: on_iteration(iteration) -> None")
         self.check = check
         self.revise = revise
         self.max_iterations = max_iterations
@@ -57,9 +70,16 @@ def __init__(
 
     # ---- streaming form: yields each iteration as it happens -------------
     def cycle(self, initial: Any) -> Iterator[Iteration]:
+        """Yield each `Iteration` as it happens.
+
+        Unlike `run()`, the streaming form does not trap stage exceptions or
+        detect plateaus — the caller owns the loop and can stop early or wrap
+        it as it sees fit. A `check` that returns something other than a
+        `Verdict` still raises a clear `TypeError`.
+        """
         state = self.seed(initial) if self.seed else initial
         for i in range(self.max_iterations):
-            verdict = self.check(state)
+            verdict = _as_verdict(self.check(state), "check", i)
             it = Iteration(index=i, state=state, verdict=verdict)
             if self.on_iteration:
                 self.on_iteration(it)
@@ -75,11 +95,21 @@ def run(self, initial: Any) -> Result:
         status = Status.EXHAUSTED
         last_improved: Optional[float] = None
         stale = 0
-        state = self.seed(initial) if self.seed else initial
+
+        # the seed is a stage too: surface its failures as ERROR, just like
+        # check/revise, rather than letting them escape the engine.
+        if self.seed is not None:
+            try:
+                state = self.seed(initial)
+            except Exception as e:  # noqa: BLE001 - surface seed errors as ERROR status
+                return Result(Status.ERROR, initial, 0, history, best,
+                              error=f"seed failed: {e}")
+        else:
+            state = initial
 
         for i in range(self.max_iterations):
             try:
-                verdict = self.check(state)
+                verdict = _as_verdict(self.check(state), "check", i)
             except Exception as e:  # noqa: BLE001 - surface stage errors as ERROR status
                 return Result(Status.ERROR, state, len(history), history, best,
                               error=f"check failed at iteration {i}: {e}")
@@ -114,6 +144,23 @@ def run(self, initial: Any) -> Result:
         return Result(status, state, len(history), history, best)
 
 
+def _as_verdict(value: Any, stage: str, index: int) -> Verdict:
+    """Guard that a stage returned a `Verdict`.
+
+    Returning a bare bool/tuple/None from `check` is the single most common
+    misuse; without this guard it surfaces deep in the engine as a cryptic
+    `AttributeError`. Here it becomes an actionable `TypeError` naming the
+    stage and iteration.
+    """
+    if not isinstance(value, Verdict):
+        raise TypeError(
+            f"{stage} must return a Verdict at iteration {index}, "
+            f"got {type(value).__name__}: {value!r}. "
+            f"Use Verdict(ok=...), Verdict.passed(...), or Verdict.failed(...)."
+        )
+    return value
+
+
 def _better(current: Optional[Iteration], candidate: Iteration) -> Iteration:
     """Keep the higher-scoring iteration; unscored candidates never displace
     a scored best, but become best if nothing better exists yet."""

demos/06_error_recovery.py

@@ -0,0 +1,64 @@
+"""Scenario 6 - resilience engineers: failures are first-class outcomes.
+
+Real refinement stages fail: a model endpoint times out, a `revise` hits a
+divide-by-zero, a `seed` preprocessor chokes on bad input. cyclework does not
+let those escape as raw tracebacks - it surfaces them as `Status.ERROR` with a
+message naming the stage and iteration, and it preserves the trace and the best
+candidate seen *before* the failure, so the caller can still recover something.
+
+This demo triggers a failure in each of the three stages (check, revise, seed)
+and shows the engine naming the stage, keeping the partial history, and handing
+back the best pre-failure candidate.
+"""
+from _common import Engine, Verdict, rule, show_summary, show_trace
+
+
+def main() -> None:
+    rule("ERROR RECOVERY  -  stage failures become inspectable outcomes")
+
+    # --- a check that explodes part-way through ---------------------------
+    print("\n1) check() raises on the 3rd call - history before it survives:\n")
+    calls = {"n": 0}
+
+    def flaky_check(s):
+        calls["n"] += 1
+        if calls["n"] == 3:
+            raise RuntimeError("scoring service 503")
+        return Verdict(ok=False, score=float(s))
+
+    res = Engine(flaky_check, lambda s, v: s + 1, max_iterations=10).run(0)
+    show_trace(res, fmt_state=str)
+    show_summary(res)
+    print(f"   -> {len(res.history)} good iterations kept; best score "
+          f"{res.best.score} from before the failure.")
+
+    # --- a revise that divides by zero ------------------------------------
+    print("\n2) revise() raises - the engine names the stage and iteration:\n")
+
+    def bad_revise(s, v):
+        return 1 / 0  # noqa: B018
+
+    res = Engine(lambda s: Verdict(ok=False, score=1.0), bad_revise,
+                 max_iterations=5).run(10)
+    show_summary(res)
+
+    # --- a seed that fails before the loop even starts --------------------
+    print("\n3) seed() raises before any iteration - clean ERROR, empty trace:\n")
+
+    def bad_seed(_x):
+        raise ValueError("could not parse initial input")
+
+    res = Engine(lambda s: Verdict(True), lambda s, v: s,
+                 seed=bad_seed).run("garbage")
+    show_summary(res)
+    print(f"   -> history length {len(res.history)} (nothing ran), "
+          f"status named the failure instead of crashing the caller.")
+
+    print(
+        "\nEvery stage failure is a value you can branch on (res.status, "
+        "res.error), not an exception that takes down the whole pipeline."
+    )
+
+
+if __name__ == "__main__":
+    main()

demos/07_gradient_descent.py

@@ -0,0 +1,58 @@
+"""Scenario 7 - ML / optimization engineers: descent as a refinement loop.
+
+Gradient descent is the archetypal "propose, check, revise" loop: the check
+scores how low the loss is, the revise steps downhill. Because cyclework tracks
+the best candidate seen, a descent that overshoots and bounces still hands back
+its lowest-loss point - not wherever it happened to stop.
+
+This demo minimizes a 1-D quadratic with three learning rates - one that
+converges smoothly, one that converges slowly (and exhausts its budget), and
+one so large it oscillates - and shows the engine naming each outcome while
+always recovering the best point.
+"""
+from _common import Engine, Status, Verdict, rule, show_summary
+
+
+def descend(lr: float, max_iterations: int = 60):
+    """Minimize f(x) = (x - 3)^2. score = -loss (higher is better)."""
+    target = 3.0
+
+    def check(x):
+        loss = (x - target) ** 2
+        return Verdict(ok=loss <= 1e-9, score=-loss, detail={"loss": loss})
+
+    def revise(x, _v):
+        grad = 2 * (x - target)      # df/dx
+        return x - lr * grad
+
+    return Engine(check, revise, max_iterations=max_iterations).run(0.0)
+
+
+def main() -> None:
+    rule("GRADIENT DESCENT  -  minimize a quadratic, keep the best point")
+    print("\nMinimize f(x) = (x - 3)^2 from x0 = 0, varying the learning rate.\n")
+
+    for lr, label in [(0.3, "well-tuned"), (0.02, "too small"), (1.05, "too large")]:
+        res = descend(lr)
+        best_x = res.best.state
+        best_loss = res.best.verdict.detail["loss"]
+        print(f"lr={lr:<5} ({label}):")
+        print(f"   status={res.status.value:<9} iterations={res.iterations:<3} "
+              f"best_x={best_x:.6f}  best_loss={best_loss:.3e}")
+        if res.status is Status.SOLVED:
+            print("   -> converged to the minimum.")
+        elif res.status is Status.EXHAUSTED:
+            print("   -> ran out of budget; best point so far is still usable.")
+        print()
+
+    print("Verbose summary of the well-tuned run:")
+    show_summary(descend(0.3))
+
+    print(
+        "\nSame engine, three regimes. The loop never loses the best candidate, "
+        "so even a non-converging run returns something you can use."
+    )
+
+
+if __name__ == "__main__":
+    main()

demos/08_text_normalizer.py

@@ -0,0 +1,80 @@
+"""Scenario 8 - data-cleaning engineers: a multi-rule normalizer.
+
+The bundled slug refiner fixes one rule per cycle; here we build a richer
+normalizer for free-text product codes from the same primitives - each cycle
+the check reports the highest-priority violation and revise applies exactly
+that fix. The point: a "clean this messy field" pipeline is a refinement loop,
+and the trace tells you *which* rules fired, in what order, for any input.
+"""
+import re
+
+from _common import Engine, Verdict, rule, show_summary, show_trace
+
+
+def normalize_code(raw: str):
+    """Normalize a product code: uppercase, A-Z0-9 and single dashes, <= 16."""
+    MAX = 16
+
+    def check(s: str) -> Verdict:
+        n = sum(c not in "-" for c in s)
+        score = float(n)  # prefer candidates closer to a valid form
+        if s != s.strip():
+            return Verdict.failed("has surrounding whitespace", score=score, fix="trim")
+        if s != s.upper():
+            return Verdict.failed("not uppercase", score=score, fix="upper")
+        if re.search(r"\s", s):
+            return Verdict.failed("internal whitespace", score=score, fix="space")
+        if re.search(r"[^A-Z0-9-]", s):
+            return Verdict.failed("illegal characters", score=score, fix="strip")
+        if "--" in s or s.startswith("-") or s.endswith("-"):
+            return Verdict.failed("dash noise", score=score, fix="dash")
+        if len(s) > MAX:
+            return Verdict.failed("too long", score=score, fix="trunc")
+        return Verdict.passed(score=score)
+
+    def revise(s: str, v: Verdict) -> str:
+        fix = v.detail.get("fix")
+        if fix == "trim":
+            return s.strip()
+        if fix == "upper":
+            return s.upper()
+        if fix == "space":
+            return re.sub(r"\s+", "-", s)
+        if fix == "strip":
+            return re.sub(r"[^A-Z0-9-]", "", s)
+        if fix == "dash":
+            return re.sub(r"-+", "-", s).strip("-")
+        if fix == "trunc":
+            return s[:MAX].rstrip("-")
+        return s
+
+    return Engine(check, revise, max_iterations=25).run(raw)
+
+
+CASES = [
+    "  sku 12 / abc!!  ",
+    "Widget---Pro---2026",
+    "VALID-CODE-01",
+    "this-is-a-really-long-product-code-that-overflows",
+]
+
+
+def main() -> None:
+    rule("TEXT NORMALIZER  -  multi-rule cleaning, one fix per cycle")
+    print("\nNormalize free-text product codes to UPPER / [A-Z0-9-] / <=16 chars.")
+    print("Each cycle names the top violation; the trace shows which rules fired.\n")
+
+    for raw in CASES:
+        print(f"input: {raw!r}")
+        res = normalize_code(raw)
+        show_trace(res, fmt_state=repr)
+        print(f"   => {res.state!r}")
+        show_summary(res)
+        print()
+
+    print("The same propose/check/revise shape scales to any rule set you can "
+          "rank by priority.")
+
+
+if __name__ == "__main__":
+    main()