From fe270838a3de3a424ec0d12e98f6049cc6108f6e Mon Sep 17 00:00:00 2001 From: Yichen Wu Date: Mon, 20 Jul 2026 20:33:29 -0700 Subject: [PATCH 1/2] Define Loop engineering strategy and v0.2 gates --- CHANGELOG.md | 10 ++ README.md | 7 + docs/README.md | 6 + docs/STRATEGY.md | 255 +++++++++++++++++++++++++++++++----- docs/comparison-openclaw.md | 3 +- 5 files changed, 249 insertions(+), 32 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a87710..638678a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ All notable changes are documented here. The project follows Semantic Versioning. +## [Unreleased] + +### Documentation + +- Defined Loop as the controlled execution layer for loop engineering, mapped the + agentic coding, developer-feedback, and external-feedback loops, and added ordered, + testable v0.2 iteration gates. +- Corrected the comparison document to acknowledge the published DeepSeek evaluation + while retaining its repeated-run and production-isolation limitations. + ## [0.1.0] - 2026-07-15 First portfolio/research release of Loop's contract-first autonomous runtime. diff --git a/README.md b/README.md index de8016e..426ac13 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,12 @@ The loop is autonomous, not unbounded. Capabilities, filesystem scope, egress po approvals, step limits, token budgets, no-progress detection, and a verification reserve are enforced by the runtime rather than left to the model's discretion. +Loop is designed as the controlled execution layer inside a broader loop-engineering +process: it owns the fast agentic coding loop, gives the developer-feedback loop +versioned evidence, and supports external feedback without pretending to replace human +product judgment. The [product strategy](./docs/STRATEGY.md) records that model and the +ordered v0.2 release gates. + ## What makes Loop different Most agent frameworks help a model call tools. Loop supervises the complete journey from @@ -216,6 +222,7 @@ docs/ ADRs, operational guides, product/system rationale ``` - [Architecture](./ARCHITECTURE.md) +- [Product strategy and v0.2 gates](./docs/STRATEGY.md) - [Verified Completion evaluation](./evals/README.md) - [Local development](./docs/guides/local-development.md) - [Deployment](./docs/guides/deployment.md) diff --git a/docs/README.md b/docs/README.md index 4b6f6e4..55bfbae 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,11 @@ # Documentation +- **[Product strategy](./STRATEGY.md)** — the three-loop product model and ordered + v0.2 release gates. +- **[Loop engine](./loop.md)** — execution invariants, verification, budgets, and + recovery behavior. +- **[Comparison](./comparison-openclaw.md)** — Loop's bounded-handoff positioning + relative to chat-first personal agents. - **[../ARCHITECTURE.md](../ARCHITECTURE.md)** — the system design in depth. - **Guides** — practical how-tos: - [Local development](./guides/local-development.md) diff --git a/docs/STRATEGY.md b/docs/STRATEGY.md index 2ca7071..e35f21a 100644 --- a/docs/STRATEGY.md +++ b/docs/STRATEGY.md @@ -2,19 +2,72 @@ ## Thesis -Loop is a contract-first autonomous execution runtime. The user delegates a goal, -authority, and budget; Loop returns artifacts plus replayable evidence. Its core -differentiator is not another chat surface or another MCP wrapper: +Loop is the controlled execution layer for loop engineering. The user delegates a +goal, authority, and budget; Loop keeps working, testing, and repairing until it can +return artifacts plus replayable evidence—or an explicit, auditable reason it could +not finish. + +Its core differentiator is not another chat surface or another MCP wrapper: > A model may propose that work is finished, but only the acceptance contract and > independently re-executed evidence can complete a strict task. +The intended experience is equally important: the user should state the goal once. +Contract construction, repository discovery, routine QA, failure diagnosis, and retry +belong inside the product rather than becoming repeated instructions the user must +write. + +## Three nested feedback loops + +Andrew Ng describes three loops for building 0-to-1 products with coding agents: +an agentic coding loop, a developer-feedback loop, and an external-feedback loop. +His earlier agentic-workflow work separates reflection, tool use, planning, and +multi-agent collaboration, and emphasizes disciplined evals and error analysis. + +This is an engineering framework rather than a claim that iteration alone guarantees +correctness. Loop adopts the framing while adding an explicit control and evidence +boundary. + +| Loop | Owner | Typical cadence | Loop's role | +| ------------------ | ----------------------- | ------------------------ | ------------------------------------------------------------------------------------------------- | +| Agentic coding | Loop runtime | Seconds to minutes | Turn a locked contract into tested artifacts and a Receipt. | +| Developer feedback | User with Loop | Tens of minutes to hours | Review the product, revise the specification, and turn discovered failures into regression evals. | +| External feedback | Users and product owner | Hours to weeks | Feed alpha usage, production behavior, and market learning back into product direction. | + +Loop should own the inner loop and make it reliable. It should make the middle loop +fast and evidence-rich. It should support the outer loop without pretending to replace +human product judgment. Humans retain a context advantage whenever they know something +about users, risk, or intent that the runtime does not. + +Primary references: + +- [Andrew Ng, “3 Key Loops for Building 0-to-1 Products with AI Agents”](https://www.linkedin.com/posts/andrewyng_loop-engineering-is-a-hot-buzzphrase-after-activity-7477753883768029185-Fg8P) +- [Agentic Design Patterns: Reflection](https://www.deeplearning.ai/the-batch/agentic-design-patterns-part-2-reflection) +- [Agentic Design Patterns: Planning](https://www.deeplearning.ai/the-batch/agentic-design-patterns-part-4-planning) +- [We Iterate on Models. We Can Iterate on Evals, Too](https://www.deeplearning.ai/the-batch/we-iterate-on-models-we-can-iterate-on-evals-too) + +## A controlled loop, not repetition + +A useful loop needs more than another model call. Every run requires six explicit +elements: + +1. **Goal:** a versioned specification and acceptance contract. +2. **State:** the workspace revision, evidence, failures, and attempted branches. +3. **Tools:** bounded ways to observe or change the world. +4. **Verifier:** executable error signals independent of the model's completion claim. +5. **Stop conditions:** success, cancellation, risk, budget, and no-progress limits. +6. **Memory:** compact decisions and evidence that improve the next action without + carrying an ever-growing transcript. + +This is why verification, authority, accounting, recovery, and observability are core +product behavior rather than infrastructure around the “real” agent. + ## Product shape Loop is an application and runtime, not merely a protocol server: -- the web/desktop surfaces collect goals, contracts, authority, and approvals; -- the API persists task state, ownership, budgets, evidence, and Receipts; +- the web/desktop surfaces collect goals, show progress, and handle material approvals; +- the API persists task state, ownership, contracts, budgets, evidence, and Receipts; - workers provide durable execution and crash recovery; - sandboxes and gateways enforce filesystem, process, provider, and network bounds; - MCP is one adapter family through which specialized capabilities may be exposed. @@ -22,44 +75,184 @@ Loop is an application and runtime, not merely a protocol server: System design matters because the product promises durable handoff under failure and least authority—not because it needs speculative internet-scale traffic. -## v0.1 release gate +## What v0.1 proved + +The v0.1 release established the narrow inner-loop foundation: + +- a fresh environment launches the zero-key verified demo with one command; +- strict tasks require mapped passing evidence and replayable Receipts; +- CI exercises the browser journey, dependency audits, desktop packaging, Redis + recovery, authority revocation, Kubernetes execution, and rollout rollback; +- one recorded DeepSeek `deepseek-chat` run solved all 12 published cases with zero + false acceptances and a passing replay; +- security boundaries, architecture, implementation, and residual risks are public. + +This proves product wiring and one real-provider sample. It does not establish broad +model quality, repeated-run confidence, production adoption, or the complete +three-loop product. + +## v0.2 flagship outcome + +The next release has one flagship journey: + +> Select a real Git repository, enter one instruction, and receive a verified patch +> that can be reviewed, applied, discarded, or undone. + +Loop must infer a sufficiently strong contract, discover the repository's checks, +work inside bounded authority, use failures to repair its work, survive interruption, +and present the final evidence without requiring the user to operate the loop. + +## Ordered iteration gates + +These are release gates, not calendar estimates. Work proceeds in this order and each +gate must be verified before the next one expands the product surface. + +### Gate 1 — One-instruction contract compiler + +- Replace the default form's manual criteria, command, artifact, capability, step, and + token configuration with a repository picker, instruction, and Run action. Keep the + existing controls in an Advanced panel. +- Compile the instruction and deterministic repository discovery into criteria, + required artifacts, regression checks, capabilities, risk, and budget. +- Run an independent contract critic that rejects tautological, non-verifiable, or + materially incomplete acceptance criteria. +- Lock the contract before the first mutation. Later executor or verifier output may + add evidence but may not weaken the locked contract. +- Continue automatically for low-risk, high-confidence work. Ask the user only when + missing context could materially change the product or authority boundary. + +**Done when:** a fresh local-project task can start from one instruction, reach a +strict verified change set, and Apply/Undo without manually authored criteria. + +#### First implementation slice + +1. Allow a local-project task to be published without user-authored criteria under a + deterministic, no-network coding capability preset. The model may recommend + additional authority but can never grant it to itself. +2. After the isolated clone exists, run read-only repository discovery before any + mutable tool call: manifests, existing quality scripts, test layout, build outputs, + and the clean baseline. +3. Add a typed `ContractDraft` containing criteria, checks, artifacts, risk, + assumptions, confidence, and any authority requests. Replace the current generic + rubric fallback for this path. +4. Ask the verifier/critic to challenge that draft. Persist and hash the accepted + contract before the executor can mutate the workspace; a material ambiguity or + authority expansion becomes `awaiting_input`. +5. Feed the same locked contract to planning, completion, Receipt generation, replay, + the task UI, and evaluation scoring so there is one acceptance source of truth. +6. Add a zero-provider deterministic test and a real-provider fixture proving the + complete instruction → contract → edit → repair → verification → change-set path. + +### Gate 2 — Cancellation, recovery, and concurrency + +- Propagate cancellation into in-flight provider calls, shell process groups, Docker + containers, Kubernetes Jobs, gateways, and delegated work. +- Inject crashes around plan persistence, tool completion, verification, queue + acknowledgement, and terminal Receipt creation. +- Stress duplicate delivery, Redis/Postgres interruption, concurrent project access, + workspace isolation, source locks, and resource admission. +- Require idempotency for external writes; do not claim exactly-once semantics where + an upstream system cannot provide them. + +**Done when:** automated tests prove prompt cancellation, safe replay/recovery, no +duplicate mutation, and no cross-workspace leakage under at least 20 concurrent tasks. + +### Gate 3 — Explicit loop state machine + +- Add characterization tests before moving behavior. +- Split the current orchestration service into explicit transition policy, decision + parsing, context budgeting, progress control, action dispatch, verification, and + delegation components. +- Persist transition reasons so recovery and the UI use the same state semantics. + +**Done when:** every allowed terminal and resumable path is covered by a transition +test, the production loop no longer depends on one monolithic service, and the existing +verified benchmark does not regress. + +### Gate 4 — Repository-level evidence and error analysis + +- Add realistic fixture repositories covering bug repair, feature work, multi-file + refactoring, CLI/API/UI changes, regressions, and incomplete specifications. +- Run every case repeatedly and publish distributions rather than a selected best run. +- Compare the same model in one-shot, tool-loop-without-gates, and full Loop modes. +- Evaluate contract quality, trajectory efficiency, tool routing, convergence, + false acceptance, artifact integrity, and replay in Docker/Kubernetes isolation. +- Turn every recurring failure class into a regression case, and revise evals when + their ranking disagrees with skilled human judgment. + +**Done when:** the primary model reaches at least 85% verified solve rate across three +runs of the repository suite, observed false acceptance remains zero, and the report +includes median/tail steps, tokens, time, questions, and stop reasons. + +### Gate 5 — Developer-feedback loop + +- Group successive deliveries into a Product Session with versioned specifications, + feedback deltas, change sets, and Receipts. +- Show evidence and visual/output differences between versions. +- Let a user turn a discovered bug into a persistent regression contract before the + next run. +- Preserve the distinction between a corrected implementation and a changed product + decision. + +**Done when:** feedback on delivery v1 produces an auditable spec v2 and verified +delivery v2 without losing the v1 contract, evidence, or rollback path. + +### Gate 6 — Isolated expert routing + +- Move Sibyl and Argus behind an authenticated, production-capable MCP gateway rather + than host subprocesses. +- Enforce per-server capability, destination, timeout, output, call, and token limits. +- Route Sibyl when a real research uncertainty blocks progress and Argus when UI or + browser evidence is required; keep both subject to no-progress controls. +- Make health, cancellation, provenance, and failures visible in the Receipt. + +**Done when:** a production-isolated research task and UI task automatically use the +appropriate expert, recover from its failure, and still satisfy the same contract and +Receipt semantics. + +### Gate 7 — Distribution and external-feedback foundation + +- Publish macOS, Windows, and Linux release artifacts with checksums; add signing, + notarization, and update metadata when credentials are available. +- Make first run validate provider access, sandbox health, repository access, and the + verified demo before accepting untrusted work. +- Add an opt-in path for a user to attach feedback or a failure report to a Product + Session without automatically modifying production code. +- Publish a short, reproducible one-instruction-to-Receipt demonstration. -A release is portfolio-ready only when all of these are reproducible: +**Done when:** a new user can install Loop, finish the flagship journey, replay its +Receipt, and submit structured feedback without repository-specific assistance. -- a fresh environment can launch the zero-key demo with one command; -- the built-in strict-contract task completes with execution verification; -- every criterion has mapped passing evidence and Receipt replay succeeds; -- CI covers the browser journey, offline unit/integration tests, dependency audits, - desktop packaging, Redis restart/worker recovery, and Kubernetes rollback; -- README, security boundaries, architecture, and implementation agree; -- a versioned release and changelog exist; -- paid real-provider results are never fabricated or implied by mock results. +## v0.2 release gate -## Priority order after v0.1 +v0.2 ships only when all of the following are reproducible: -1. **Evidence:** run the published suite across selected real models; report solve - rate, false acceptances, tokens, steps, and wall time. -2. **Core reliability:** expand adversarial contracts, replay portability, crash and - concurrency tests, and reduce orchestration complexity where it obscures invariants. -3. **Isolation:** move local Sibyl/Argus integrations behind a production-capable - isolated MCP gateway and make browser sessions recoverable or explicitly disposable. -4. **Distribution:** sign/notarize desktop installers, publish upgrade guidance, and - earn real external usage. -5. **Breadth:** add integrations only when they preserve the same authority, approval, - evidence, and Receipt semantics. +- the flagship local-repository journey needs one user instruction for routine work; +- generated contracts are locked, independently criticized, and execution-verifiable; +- cancellation, crash recovery, duplicate delivery, and concurrent isolation pass; +- repeated repository-level results and all failures are published honestly; +- a developer-feedback revision retains both specifications and both Receipts; +- release artifacts and a clean-machine first-run path exist; +- README, security boundaries, architecture, evaluation reports, and behavior agree. ## Non-goals - Claiming that prompt injection or semantic failure is impossible. -- Matching another agent's channel or skill count before the flagship path is proven. +- Removing human approval when risk or missing context materially affects the outcome. +- Matching another agent's channel, skill, or sub-agent count before the flagship path + is proven. +- Letting free-form planning or multi-agent conversation replace deterministic checks. - Building speculative scaling layers that are not exercised by acceptance tests. - Treating a green unit suite as proof that the documented user journey works. ## Success metrics -- verified solve rate and false-acceptance rate by suite/model/revision; -- median and tail tokens, steps, and wall time; -- Receipt replay pass rate after process restart; +- verified solve and false-acceptance rates by suite, model, isolation, and revision; +- median and tail tokens, steps, questions, tool calls, and wall time; +- contract-critic rejection and human contract-correction rates; +- no-progress stops and useful recovery after a blocked branch; +- Receipt replay after process restart and across a clean environment; - task recovery after worker loss and duplicate delivery; -- first-run demo success from a clean environment; +- first-run flagship success from a clean installation; +- developer-feedback cycle time and regressions captured from feedback; - external users who reproduce a Receipt rather than stars alone. diff --git a/docs/comparison-openclaw.md b/docs/comparison-openclaw.md index c295da8..909411b 100644 --- a/docs/comparison-openclaw.md +++ b/docs/comparison-openclaw.md @@ -39,7 +39,8 @@ injection is impossible. ## Where Loop remains behind - It has little adoption history and no large third-party extension ecosystem. -- Public real-provider benchmark results have not yet been paid for and published. +- It has one published 12-case DeepSeek run, but no repeated-run, cross-model, or + production-isolation benchmark yet. - Desktop installers are CI-built but not publicly signed, notarized, or auto-updated. - The safest local path depends on Docker; the zero-key demo uses visibly reduced inline isolation. From ae1304bdba5778c9c5984a71bb3e74d15a310422 Mon Sep 17 00:00:00 2001 From: Yichen Wu Date: Mon, 20 Jul 2026 20:39:13 -0700 Subject: [PATCH 2/2] Pin patched brace-expansion releases --- CHANGELOG.md | 4 ++++ pnpm-lock.yaml | 18 ++++++++++-------- pnpm-workspace.yaml | 2 ++ 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 638678a..b0fa5fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ All notable changes are documented here. The project follows Semantic Versioning ## [Unreleased] +### Security + +- Pinned patched `brace-expansion` releases across transitive dependency trees. + ### Documentation - Defined Loop as the controlled execution layer for loop engineering, mapped the diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 4a06611..f02b0c1 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -6,6 +6,8 @@ settings: overrides: '@electron/rebuild': 4.2.0 + brace-expansion@<1.1.16: 1.1.16 + brace-expansion@>=3.0.0 <5.0.7: 5.0.7 esbuild: 0.28.1 postcss: 8.5.19 tmp: 0.2.7 @@ -1955,14 +1957,14 @@ packages: resolution: {integrity: sha512-d0II/GO9uf9lfUHH2BQsjxzRJZBdsjgsBiW4BvhWk/3qoKwQFjIDVN19PfX8F2D/r9PCMTtLWjYVCFrpeYUzsw==} deprecated: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info. - brace-expansion@1.1.15: - resolution: {integrity: sha512-EwOCDEex4quD37XhqM3omwtMoJjr//isUZz1JopUNWms+4Z2ViyM/k1YIRePpoVNnQhENnxtFjLaxNHrT7xIUg==} + brace-expansion@1.1.16: + resolution: {integrity: sha512-IDw48K2/2kRkg9LdJxurvq3lV3aBgq0REY89duEqFRthjlPdXHKMj7EnQOXVckxzgisinf3nHfrcE2FufFLXMw==} brace-expansion@2.1.1: resolution: {integrity: sha512-WR1cURNjuvBLMZBMbqM0UoE+WAfdUcEV1ccD8PVBVOI+Z3ND4+SZbN8RsfT2bMuG1qwz5RFvPukSZm5fF2D5eA==} - brace-expansion@5.0.6: - resolution: {integrity: sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g==} + brace-expansion@5.0.7: + resolution: {integrity: sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==} engines: {node: 18 || 20 || >=22} braces@3.0.3: @@ -6340,7 +6342,7 @@ snapshots: boolean@3.2.0: optional: true - brace-expansion@1.1.15: + brace-expansion@1.1.16: dependencies: balanced-match: 1.0.2 concat-map: 0.0.1 @@ -6349,7 +6351,7 @@ snapshots: dependencies: balanced-match: 1.0.2 - brace-expansion@5.0.6: + brace-expansion@5.0.7: dependencies: balanced-match: 4.0.4 @@ -7887,11 +7889,11 @@ snapshots: minimatch@10.2.5: dependencies: - brace-expansion: 5.0.6 + brace-expansion: 5.0.7 minimatch@3.1.5: dependencies: - brace-expansion: 1.1.15 + brace-expansion: 1.1.16 minimatch@5.1.9: dependencies: diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 789950c..9671b51 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -12,6 +12,8 @@ peerDependencyRules: overrides: '@electron/rebuild': '4.2.0' + 'brace-expansion@<1.1.16': '1.1.16' + 'brace-expansion@>=3.0.0 <5.0.7': '5.0.7' esbuild: '0.28.1' postcss: '8.5.19' tmp: '0.2.7'