Add repository evaluation matrix

.env.example

@@ -54,6 +54,8 @@ GLM_API_KEY=
 OLLAMA_BASE_URL=                    # e.g. http://localhost:11434
 OLLAMA_MODEL=llama3.2
 LLM_DEFAULT_PROVIDER=deepseek       # anthropic | deepseek | gemini | glm | ollama
+LLM_TIMEOUT_SECONDS=120             # timeout for one provider request
+LLM_TOTAL_TIMEOUT_SECONDS=180       # hard deadline across retries and fallbacks
 # A retryable failure (timeout / 5xx / empty) is retried on the same provider this
 # many times (with linear backoff) before cascading to the next one.
 LLM_MAX_RETRIES=2
@@ -70,7 +72,7 @@ AGENT_MAX_STEPS_DEFAULT=12
 AGENT_MAX_STEPS_CAP=40
 LOOP_TOKEN_BUDGET_DEFAULT=60000
 LOOP_TOKEN_BUDGET_CAP=200000
-AGENT_VERIFICATION_TOKEN_RESERVE=16000
+AGENT_VERIFICATION_TOKEN_RESERVE=6000
 AGENT_STUCK_THRESHOLD=4
 AGENT_REPEATED_ACTION_LIMIT=2
 AGENT_EXPLORATION_BRANCH_CAP=8

.gitignore

@@ -14,6 +14,8 @@ apps/web/test-results/
 /evals/results/*
 !/evals/results/demo-smoke.json
 !/evals/results/deepseek-chat-v0.1.0.json
+!/evals/results/deepseek-chat-repository-matrix-v0.2.0.json
+!/evals/results/deepseek-chat-full-loop-v0.2.12.json
 
 # ---------- Python ----------
 __pycache__/

CHANGELOG.md

@@ -17,10 +17,49 @@ All notable changes are documented here. The project follows Semantic Versioning
   deduplication of deterministic repository checks.
 - Made discovered Python test gates use `python -m pytest -q` so flat repositories run
   consistently across virtual environments and pytest entrypoint modes.
+- Added bounded source/test previews to repository discovery, deterministic test-gate
+  canonicalization when every criterion is grounded in test-source evidence, validation
+  for malformed inline Python and subprocess-output checks, and safe adjudication of
+  non-user-answerable critic noise.
+- Reduced repeated-file inspection and evidence-free branches, compacted planner history,
+  lowered planner variance, reserved a right-sized final-verification budget, and fed
+  automatic contract failures directly into the next decision.
+- Added one total deadline across provider retries/fallbacks so a failing model request
+  cannot multiply the configured timeout across every route.
+- Added integer-or-`nonzero` exit contracts and consistent replay semantics for negative
+  command assertions.
+
+### Reliability
+
+- Made the local in-memory rate limiter preserve the original fixed-window expiry, matching
+  Redis behavior.
+- Added atomic repository-matrix checkpoints that reject manifest, fixture, runtime, mode,
+  or selection drift, plus bounded retry of task publication throttling.
+- Expanded no-progress recovery so new failure output is useful evidence while repeated
+  output, immediate post-write rereads, duplicate research calls, and unchanged finish
+  attempts remain bounded.
+
+### Evaluation
+
+- Added eight protected fixture repositories spanning bug repair, feature work,
+  multi-file refactoring, CLI, API, UI, regression preservation, and incomplete
+  specifications.
+- Added a three-mode evaluator for one-shot, ungated tool loop, and full Loop with
+  protected-test hashes, independent double oracle execution, artifact and source
+  integrity checks, Receipt replay, Apply/Undo, trajectory taxonomy, and distribution
+  reporting.
+- Published the frozen DeepSeek `deepseek-chat` full-Loop matrix: 20/21 deliverable
+  attempts solved (95.24%), 3/3 safe specification deferrals, zero false acceptances,
+  and one disclosed fail-closed contract-compilation failure.
+- Archived the earlier same-model comparison and its exact v0.1 manifest; versioned the
+  corrected configuration fixture instead of presenting an eval-spec change as a pure
+  product improvement.
 
 ### Security
 
 - Pinned patched `brace-expansion` releases across transitive dependency trees.
+- Pinned `sharp` 0.35.3 to eliminate inherited libvips vulnerabilities and moved the
+  workspace off the broken pnpm 11.13.0 release.
 
 ### Documentation
 
@@ -31,6 +70,8 @@ All notable changes are documented here. The project follows Semantic Versioning
   while retaining its repeated-run and production-isolation limitations.
 - Published the first real-provider one-instruction local-project result, including
   contract locking, execution verification, Receipt replay, Apply, and Undo evidence.
+- Documented the repository-level protocol, repeated-run distributions, historical
+  comparison, current limitations, and the remaining Docker/Kubernetes Gate 4 subgate.
 
 ## [0.1.0] - 2026-07-15
 

README.md

@@ -129,6 +129,36 @@ own contract, changed the implementation and tests, completed in 5 steps using 8
 provider-reported tokens and 13.459 seconds, then passed Receipt replay, Apply, and
 Undo. This is one clean Gate 1 sample, not a repeated-run reliability estimate.
 
+## Repository-level evidence
+
+The current [repository matrix](./evals/repository-suite.json) exercises eight small
+repositories across bug repair, feature work, multi-file refactoring, CLI, API, UI,
+regression preservation, and a materially contradictory specification. Each case runs
+three times. Protected tests are hash-checked, external oracles run twice on independent
+copies, accepted patches must survive Receipt replay and Apply/Undo, and a successful
+report requires at least an 85% verified solve rate with zero false acceptances.
+
+The frozen [DeepSeek `deepseek-chat` full-Loop report](./evals/results/deepseek-chat-full-loop-v0.2.12.json)
+completed all 24 attempts. Loop solved 20/21 deliverable attempts (**95.24%**), safely
+deferred all 3 contradictory-spec attempts, and recorded **0 false acceptances**.
+Median/p95/max were 4/7/9 steps, 10,386/19,736/25,656 provider-reported tokens, and
+16.011/22.589/27.213 seconds. The one unsolved attempt was an unnecessary UI
+clarification after the contract compiler returned an invalid empty criteria list; Loop
+failed closed before mutation rather than accepting unverified work.
+
+An [archived same-model comparison](./evals/results/deepseek-chat-repository-matrix-v0.2.0.json)
+also records one-shot, ungated tool loop, and full Loop on the versioned v0.1 matrix.
+One-shot was cheap but produced 3 false acceptances; the ungated loop solved 20/21
+without a false acceptance; the earlier full Loop solved 17/21 with none. That report
+guided the contract, context, and convergence changes above. It is not presented as a
+direct before/after score because the ambiguous configuration fixture was corrected and
+versioned before the final run.
+
+Both repository reports used macOS `inline` execution, visibly reduced isolation, and
+one model. They establish repeated repository-level evidence for the control loop, not
+cross-model confidence or Docker/Kubernetes sandbox quality. See the
+[evaluation protocol](./evals/README.md) for reproduction and limitations.
+
 ## The trust boundary
 
 Loop treats the model as a planner, not an authority source.
@@ -252,8 +282,9 @@ docs/              ADRs, operational guides, product/system rationale
 ## Status
 
 `v0.1.0` is a serious portfolio/research release, not a claim of production mileage
-or broad adoption. The narrow verified coding path is automated end-to-end; provider
-quality still depends on the selected model and should be evaluated with the published
-suite before trusting a workload.
+or broad adoption. The one-instruction repository path now clears its repeated inline
+evidence threshold, while one fail-closed contract-compilation miss and the production
+isolation rerun remain explicit work. Provider quality still depends on the selected
+model and should be evaluated with the published suite before trusting a workload.
 
 MIT licensed.

apps/api/app/cache/redis.py

@@ -82,9 +82,17 @@ async def delete(self, key: str) -> None:
         self._store.pop(key, None)
 
     async def incr(self, key: str, *, ttl_seconds: int | None = None) -> int:
-        current = 0 if self._expired(key) else int(self._store[key][0])
+        expired = self._expired(key)
+        current = 0 if expired else int(self._store[key][0])
         current += 1
-        await self.set(key, str(current), ttl_seconds=ttl_seconds)
+        expires_at = (
+            time.monotonic() + ttl_seconds
+            if expired and ttl_seconds
+            else None
+            if expired
+            else self._store[key][1]
+        )
+        self._store[key] = (str(current), expires_at)
         return current
 
     async def close(self) -> None:

apps/api/app/core/config.py

@@ -117,13 +117,13 @@ def parse_cors_origins(cls, value: object) -> object:
         Literal["anthropic", "deepseek", "gemini", "glm", "ollama", "mock"] | None
     ) = None
     llm_timeout_seconds: int = 120
+    llm_total_timeout_seconds: float = Field(default=180.0, ge=1.0)
     # Retry a retryable failure (timeout, 5xx, empty) on the same provider before
     # cascading — one transient blip shouldn't fail a whole task. A mid-run failure
     # is expensive: it discards a partially-complete run (the model may already have
     # written correct output), so the budget is set to ride out a multi-second
-    # overload of a reasoning model, not just an instant blip. With these defaults:
-    # 5 attempts, linear backoff summing to ~7.5s. Bounded so a negative value can't
-    # make complete() skip every provider (client asserts >=1 attempt).
+    # overload of a reasoning model, not just an instant blip. The total deadline
+    # bounds the complete cascade even when individual requests time out slowly.
     llm_max_retries: int = Field(default=4, ge=0)
     llm_retry_backoff_seconds: float = Field(default=0.75, ge=0.0)
 
@@ -237,7 +237,7 @@ def authority_signing_key_pem(self) -> str | None:
     agent_max_spawn_depth: int = 2  # how deep sub-agents may delegate further
     agent_repeated_action_limit: int = Field(default=2, ge=1, le=10)
     agent_exploration_branch_cap: int = Field(default=8, ge=1, le=20)
-    agent_verification_token_reserve: int = Field(default=16_000, ge=500)
+    agent_verification_token_reserve: int = Field(default=6_000, ge=500)
 
     # ---- Trigger heartbeat (scheduled firing) ----
     scheduler_enabled: bool = True

apps/api/app/core/llm/client.py

@@ -56,6 +56,8 @@ async def complete(
         last_error: LLMError | None = None
         spent = 0
         input_bound = _token_estimate(system, user)
+        loop = asyncio.get_running_loop()
+        deadline = loop.time() + settings.llm_total_timeout_seconds
 
         for index, provider in enumerate(chain):
             adapter = PROVIDERS[provider].adapter
@@ -72,6 +74,9 @@ async def complete(
             # Retry this provider on a transient error before cascading, so one
             # blip (timeout/5xx/empty) doesn't fail the task on a single-provider setup.
             for attempt in range(settings.llm_max_retries + 1):
+                remaining_seconds = deadline - loop.time()
+                if remaining_seconds <= 0:
+                    raise _total_deadline_error(spent)
                 remaining = None if token_budget is None else token_budget - spent
                 if remaining is not None and remaining <= input_bound:
                     raise LLMError(
@@ -83,14 +88,15 @@ async def complete(
                 if remaining is not None:
                     attempt_max = min(max_tokens, remaining - input_bound)
                 try:
-                    content, tokens = await adapter(
-                        self._client,
-                        api_key,
-                        system,
-                        user,
-                        max_tokens=attempt_max,
-                        temperature=temperature,
-                    )
+                    async with asyncio.timeout(remaining_seconds):
+                        content, tokens = await adapter(
+                            self._client,
+                            api_key,
+                            system,
+                            user,
+                            max_tokens=attempt_max,
+                            temperature=temperature,
+                        )
                     if not content.strip():
                         raise LLMError(f"{provider} returned empty content", retryable=True)
                     if tokens:
@@ -107,6 +113,9 @@ async def complete(
                         fallbacks=fallbacks,
                         model=provider_model(provider),
                     )
+                except TimeoutError:
+                    spent += input_bound + attempt_max
+                    raise _total_deadline_error(spent) from None
                 except LLMError as exc:
                     last_error = exc
                 except Exception as exc:  # unexpected shape -> normalise, maybe cascade
@@ -123,7 +132,11 @@ async def complete(
                     ) from last_error
                 if attempt < settings.llm_max_retries:
                     log.info("llm.retry", provider=provider, attempt=attempt + 1)
-                    await asyncio.sleep(settings.llm_retry_backoff_seconds * (attempt + 1))
+                    backoff = settings.llm_retry_backoff_seconds * (attempt + 1)
+                    remaining_seconds = deadline - loop.time()
+                    if remaining_seconds <= 0:
+                        raise _total_deadline_error(spent)
+                    await asyncio.sleep(min(backoff, remaining_seconds))
 
         assert last_error is not None
         last_error.tokens_spent += spent
@@ -155,6 +168,14 @@ def _failed_attempt_charge(error: LLMError, input_bound: int, output_bound: int)
     return input_bound + (output_bound if output_may_have_been_generated else 0)
 
 
+def _total_deadline_error(tokens_spent: int) -> LLMError:
+    return LLMError(
+        "LLM call exceeded its total timeout across retries and provider fallbacks",
+        retryable=True,
+        tokens_spent=tokens_spent,
+    )
+
+
 def get_llm_client() -> FallbackLLMClient:
     """Process-wide singleton so the HTTP connection pool is reused."""
     global _client

apps/api/app/schemas/contract.py

@@ -15,7 +15,7 @@ class ContractCheck(BaseModel):
     command: str | None = Field(default=None, max_length=1_000)
     path: str | None = Field(default=None, max_length=500)
     text: str | None = Field(default=None, max_length=2_000)
-    expect_exit: int = 0
+    expect_exit: int | Literal["nonzero"] = 0
     expect_stdout: str | None = Field(default=None, max_length=2_000)
     criterion_ids: list[str] = Field(default_factory=list, max_length=12)
     source: Literal["contract", "system"] = "contract"
@@ -48,14 +48,18 @@ class RepositoryDiscovery(BaseModel):
     test_files: list[str] = Field(default_factory=list, max_length=100)
     build_outputs: list[str] = Field(default_factory=list, max_length=50)
     quality_checks: list[ContractCheck] = Field(default_factory=list, max_length=16)
+    file_previews: dict[str, str] = Field(default_factory=dict, max_length=24)
     files_scanned: int = Field(default=0, ge=0)
     truncated: bool = False
+    previews_truncated: bool = False
 
 
 class ContractCritique(BaseModel):
     accepted: bool
     issues: list[str] = Field(default_factory=list, max_length=12)
     question: str | None = Field(default=None, max_length=1_000)
+    adjudicated: bool = False
+    adjudication_reason: str | None = Field(default=None, max_length=1_000)
     provider: str = Field(default="unknown", max_length=80)
     model: str = Field(default="unknown", max_length=160)
 

apps/api/app/services/agent_react.py

@@ -164,6 +164,7 @@ def __init__(
         self._browser_specs = ""  # MCP browser tool list, injected into planning
         self._mcp_specs = ""
         self._notices = ""  # run-time notices for the planner (e.g. a tool went missing)
+        self._contract_notice = ""
         self._email_specs = ""  # email tool list, injected into planning
         self._calendar_specs = ""  # calendar tool list, injected into planning
         self._vision_specs = ""  # see_image tool, injected into planning when available
@@ -379,6 +380,7 @@ async def _run_claimed(self, task_id: uuid.UUID) -> None:
         self._calendar_specs = ""
         self._vision_specs = ""
         self._notices = ""
+        self._contract_notice = ""
         if task.skill:
             store = SkillStore(Path(settings.agent_skills_root), settings.trust_public_key_pem())
             skill = store.load(task.skill)
@@ -1229,23 +1231,26 @@ async def _run_completion_checks(
             infer_criterion_ids=task.verification_mode != "strict",
         )
 
-    async def _contract_evidence_ready(self, task: TaskModel, workspace: Workspace) -> bool:
+    async def _contract_evidence_status(
+        self, task: TaskModel, workspace: Workspace
+    ) -> tuple[bool, str]:
         contract_checks = [
             check for check in (task.required_checks or []) if check.get("source") == "contract"
         ]
         if not contract_checks:
-            return False
+            return False, ""
         checks = merge_completion_checks(
             contract_checks,
             [],
             criterion_count=len(task.rubric or []),
         )
         results = await self._run_completion_checks(task, workspace, checks)
-        return (
+        ready = (
             bool(results)
             and completion_gates_pass(results)
             and execution_coverage_complete(results, len(task.rubric or []))
         )
+        return ready, checks_summary(results)
 
     async def _start_browser(self, envelope: CapabilityEnvelope) -> McpBrowser | None:
         if not (
@@ -1411,7 +1416,7 @@ async def _run_loop(
                 self._calendar_specs,
                 self._vision_specs,
                 self._conversation,
-                notices=self._notices,
+                notices=self._notices + self._contract_notice,
                 allow_spawn=task.depth < settings.agent_max_spawn_depth,
                 today=date.today().isoformat(),
                 progress_state=guard.state(verification_reserve),
@@ -1423,7 +1428,7 @@ async def _run_loop(
                     system,
                     user,
                     max_tokens=_PLAN_MAX_TOKENS,
-                    temperature=0.5,
+                    temperature=0.1,
                     token_budget=planning_budget,
                 )
             except LLMError as exc:
@@ -1696,12 +1701,23 @@ async def _run_loop(
             auto_finish_candidate = (
                 tool in {"write_file", "edit_file"} and workspace_changed
             ) or tool == "run_command"
-            if (
-                auto_finish_candidate
-                and status is ToolStatus.OK
-                and number < task.max_steps
-                and await self._contract_evidence_ready(task, workspace)
-            ):
+            evidence_ready = False
+            if auto_finish_candidate and status is ToolStatus.OK and number < task.max_steps:
+                evidence_ready, evidence_summary = await self._contract_evidence_status(
+                    task, workspace
+                )
+                self._contract_notice = (
+                    "\nLatest automatic contract check after the previous action:\n"
+                    f"[DATA] {evidence_summary[:3_000]}\n"
+                    "Use this failure directly. After an explicit failed check, re-read at most "
+                    "one changed file only if exact context is needed for the corrective edit; "
+                    "do not re-explore unchanged files.\n"
+                    if evidence_summary and not evidence_ready
+                    else ""
+                )
+                if evidence_summary and not evidence_ready and tool != "write_file":
+                    guard.permit_diagnostic_reinspection()
+            if auto_finish_candidate and status is ToolStatus.OK and evidence_ready:
                 auto_number = number + 1
                 auto_args = {
                     "summary": (

apps/api/app/services/completion.py

@@ -27,7 +27,7 @@ def discover_project_checks(root: Path) -> list[dict[str, Any]]:
     pyproject_text = pyproject.read_text(errors="replace") if pyproject.is_file() else ""
     has_python_tests = (root / "tests").is_dir() or any(root.glob("test_*.py"))
     if has_python_tests:
-        checks.append(_command_check("system-python-test", "python -m pytest -q"))
+        checks.append(_command_check("system-python-test", "python3 -m pytest -q"))
     if "[tool.ruff" in pyproject_text:
         checks.append(_command_check("system-python-lint", "ruff check ."))
     if "[tool.mypy" in pyproject_text: