diff --git a/README.md b/README.md index 1910d75..de8016e 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,8 @@ # Loop -**Hand off a goal and an acceptance contract. Loop works in isolation, re-runs -the evidence, and returns a Receipt anyone can replay.** +**Give Loop one instruction. It keeps working, testing, and repairing its own +result until it can prove the work is ready to deliver.** [![CI](https://github.com/chriswu727/loop-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/chriswu727/loop-agent/actions/workflows/ci.yml) [![Desktop](https://github.com/chriswu727/loop-agent/actions/workflows/desktop.yml/badge.svg)](https://github.com/chriswu727/loop-agent/actions/workflows/desktop.yml) @@ -18,36 +18,60 @@ the evidence, and returns a Receipt anyone can replay.** A completed Loop task with re-executed checks, a verified Receipt, and downloadable artifacts

-## What makes Loop different +## The core idea -Most coding agents end when the model says it is done. Loop separates **work** from -**acceptance**: +Using an autonomous agent should not mean repeatedly telling it to continue, checking +whether it really ran the tests, or discovering that “done” meant “I wrote some code.” +The user should state the goal once. Loop owns the work between that instruction and a +verified delivery. -1. You confirm concrete success criteria, required final artifacts, and optional exact - verification commands. -2. Loop works inside a per-task workspace under a server-enforced capability and - token/step envelope. -3. An independent verifier re-runs each check on its own fresh workspace snapshot. -4. Every criterion must map to passing execution evidence in strict mode. -5. Loop emits a content-addressed Receipt with the contract, checks, model/runtime - provenance, output hashes, and the head of a hash-chained step ledger. -6. The Receipt can be replayed later through the API, CLI, or bundled GitHub Action. +Loop turns that idea into a bounded execution protocol: -If the evidence fails, the task does not become `completed` merely because the model -called `finish`. Limit, stuck, cancelled, and error outcomes still receive an -explicitly unverified Receipt for auditability. +1. Translate the instruction into a concrete acceptance contract. +2. Plan one useful action, execute it, and observe the real result. +3. Feed failures back into the next decision instead of handing them to the user. +4. Repeat until the required artifacts exist and every acceptance check passes. +5. Re-run the evidence independently on fresh workspace snapshots. +6. Deliver the artifacts with a replayable Receipt—or return an explicit, auditable + failure when the limits are exhausted. ```mermaid flowchart LR - G(["Goal + contract"]) --> P["Plan one action"] - P --> A["Act inside authority envelope"] - A --> O["Observe"] - O --> P - P -. "finish claim" .-> V{"Re-run checks\non a fresh copy"} - V -. "fails" .-> P - V == "passes + full coverage" ==> R(["Replayable Receipt"]) + U(["One user instruction"]) --> C["Acceptance contract"] + C --> P["Plan one action"] + P --> A["Act inside bounded authority"] + A --> O["Observe real output"] + O --> V{"Independent checks pass?"} + V -- "No: failure becomes evidence" --> P + V -- "Yes" --> D(["Verified delivery + Receipt"]) ``` +The loop is autonomous, not unbounded. Capabilities, filesystem scope, egress policy, +approvals, step limits, token budgets, no-progress detection, and a verification reserve +are enforced by the runtime rather than left to the model's discretion. + +## What makes Loop different + +Most agent frameworks help a model call tools. Loop supervises the complete journey from +an instruction to an accepted result. It separates **work** from **acceptance**: + +- **The model proposes; the runtime decides.** Tool access and resource limits come from + a server-enforced authority envelope. +- **A claim is not completion.** Strict tasks complete only when every criterion maps to + passing execution evidence and every required artifact is present. +- **Failure drives the next loop.** Test output, verifier feedback, and blocked actions are + preserved as evidence while repeated or evidence-free branches are stopped. +- **Verification is isolated.** Each check runs on its own fresh workspace snapshot, so + one check cannot manufacture state for another. +- **Delivery is auditable.** The final Receipt records the contract, checks, model and + runtime provenance, output hashes, and the head of a hash-chained step ledger. +- **Proof is portable.** The Receipt can be replayed later through the API, CLI, or bundled + GitHub Action. + +If the evidence fails, the task does not become `completed` merely because the model +called `finish`. Limit, stuck, cancelled, and error outcomes still receive an +explicitly unverified Receipt for auditability. + ## Run the verified demo Requirements: Node 22.13+, pnpm 11+ (or Corepack), and Python 3.12+. No provider @@ -82,7 +106,7 @@ It proves product wiring, not general model quality. A recorded [DeepSeek `deepseek-chat` run](./evals/results/deepseek-chat-v0.1.0.json) of the [12-case real-provider suite](./evals/verified-completion.json) solved `12/12` -with zero false acceptances: 39 steps, 64,447 provider-reported tokens, and 94.954 +with zero false acceptances: 30 steps, 42,403 provider-reported tokens, and 65.795 seconds. Every case was execution-verified, fully covered, artifact-complete, integrity-valid, and replayable. This is one clean local run using visibly reduced `inline` isolation—not a cross-model quality claim or a production-sandbox result. diff --git a/apps/api/app/services/agent_react.py b/apps/api/app/services/agent_react.py index a289018..affe733 100644 --- a/apps/api/app/services/agent_react.py +++ b/apps/api/app/services/agent_react.py @@ -58,6 +58,7 @@ from app.services.completion import ( attach_baseline, completion_gates_pass, + declared_contract_coverage_complete, discover_project_checks, mark_supplementary_agent_checks, merge_completion_checks, @@ -1380,10 +1381,12 @@ async def _run_loop( consecutive_no_progress = guard.no_progress workspace_changed = workspace.state_marker() != workspace_before + auto_finish_candidate = ( + tool in {"write_file", "edit_file"} and workspace_changed + ) or tool == "run_command" if ( - tool in {"write_file", "edit_file"} + auto_finish_candidate and status is ToolStatus.OK - and workspace_changed and number < task.max_steps and await self._contract_evidence_ready(task, workspace) ): @@ -1760,11 +1763,20 @@ async def _handle_finish( if isinstance(raw_checks, list) else [] ) - checks = merge_completion_checks( + required_checks = merge_completion_checks( task.required_checks or [], - proposed_checks, + [], criterion_count=len(task.rubric or []), ) + checks = ( + required_checks + if declared_contract_coverage_complete(required_checks, len(task.rubric or [])) + else merge_completion_checks( + required_checks, + proposed_checks, + criterion_count=len(task.rubric or []), + ) + ) check_results = attach_baseline( await self._run_completion_checks(task, workspace, checks), task.baseline_checks or [], @@ -1996,15 +2008,26 @@ def _history_view(self) -> str: compacted into durable decisions/evidence so a long run stays bounded without forgetting failed branches.""" if len(self._history) <= _HISTORY_WINDOW: - return "\n".join(entry.render() for entry in self._history) or "(nothing yet)" + rendered = [entry.render() for entry in self._history[:-1]] + if self._history: + rendered.append(self._render_latest_history(self._history[-1])) + return "\n".join(rendered) or "(nothing yet)" older = self._history[:-_HISTORY_WINDOW] recent = self._history[-_HISTORY_WINDOW:] return ( compact_history(older) + "\n\n[RECENT STEPS]\n" - + "\n".join(entry.render() for entry in recent) + + "\n".join( + self._render_latest_history(entry) if index == len(recent) - 1 else entry.render() + for index, entry in enumerate(recent) + ) ) + @staticmethod + def _render_latest_history(entry: HistoryEntry) -> str: + limit = 2_400 if entry.tool in {"run_command", "read_file"} else None + return entry.render(observation_limit=limit) + @staticmethod def _required_checks_view(checks: list[dict[str, Any]]) -> str: lines = [] diff --git a/apps/api/app/services/completion.py b/apps/api/app/services/completion.py index 5639ed7..a524f0e 100644 --- a/apps/api/app/services/completion.py +++ b/apps/api/app/services/completion.py @@ -67,6 +67,19 @@ def merge_completion_checks( return list(merged.values()) +def declared_contract_coverage_complete(checks: list[dict[str, Any]], criterion_count: int) -> bool: + if criterion_count <= 0: + return False + expected = {f"criterion-{index:03d}" for index in range(1, criterion_count + 1)} + covered = { + str(criterion) + for check in checks + if check.get("source") == "contract" + for criterion in check.get("criterion_ids", []) + } + return expected <= covered + + def attach_baseline( results: list[CheckResult], baseline: list[dict[str, Any]] ) -> list[CheckResult]: diff --git a/apps/api/app/services/evaluation.py b/apps/api/app/services/evaluation.py index f8ae71c..8caf4ee 100644 --- a/apps/api/app/services/evaluation.py +++ b/apps/api/app/services/evaluation.py @@ -67,17 +67,19 @@ def aggregate_verified_completion(results: list[dict[str, Any]]) -> dict[str, An false_acceptances = sum(bool(item.get("false_acceptance")) for item in results) total_steps = sum(int(item.get("steps_used", 0)) for item in results) total_tokens = sum(int(item.get("tokens_used", 0)) for item in results) - total_duration_seconds = sum(float(item.get("duration_seconds", 0)) for item in results) + total_duration_seconds = round( + sum(float(item.get("duration_seconds", 0)) for item in results), 3 + ) return { "cases": total, "solved": solved, - "solve_rate": solved / total if total else 0.0, + "solve_rate": round(solved / total, 4) if total else 0.0, "false_acceptances": false_acceptances, - "false_acceptance_rate": false_acceptances / total if total else 0.0, + "false_acceptance_rate": round(false_acceptances / total, 4) if total else 0.0, "total_steps": total_steps, "total_tokens": total_tokens, "total_duration_seconds": total_duration_seconds, - "average_steps": total_steps / total if total else 0.0, - "average_tokens": total_tokens / total if total else 0.0, - "average_duration_seconds": total_duration_seconds / total if total else 0.0, + "average_steps": round(total_steps / total, 3) if total else 0.0, + "average_tokens": round(total_tokens / total, 3) if total else 0.0, + "average_duration_seconds": round(total_duration_seconds / total, 3) if total else 0.0, } diff --git a/apps/api/app/services/progress.py b/apps/api/app/services/progress.py index a91e34d..8404fbd 100644 --- a/apps/api/app/services/progress.py +++ b/apps/api/app/services/progress.py @@ -25,13 +25,16 @@ class HistoryEntry: observation: str status: ToolStatus - def render(self) -> str: + def render(self, *, observation_limit: int | None = None) -> str: arg_preview = ", ".join(f"{key}={str(value)[:60]!r}" for key, value in self.args.items()) - observation_limit = 1_600 if self.tool.startswith(("sibyl_", "argus_", "browser_")) else 600 + if observation_limit is None: + observation_limit = ( + 1_600 if self.tool.startswith(("sibyl_", "argus_", "browser_")) else 600 + ) obs = ( self.observation if len(self.observation) <= observation_limit - else self.observation[:observation_limit] + " …[truncated]" + else self.observation[:observation_limit] + " …[observation preview truncated]" ) return ( f"Step {self.number} [{self.tool}] ({self.status.value}): {self.thought}\n" diff --git a/apps/api/app/services/prompts.py b/apps/api/app/services/prompts.py index 9e06a60..aad46ec 100644 --- a/apps/api/app/services/prompts.py +++ b/apps/api/app/services/prompts.py @@ -95,9 +95,13 @@ def plan_prompts( "action should usually run it (run_command) — never rewrite the same file " "twice in a row without running it in between.\n" "- Never read_file — or re-inspect with cat/head/tail/wc via run_command — a " - "file you just wrote; write_file already echoed its full contents. Spend steps " - "on progress (run it, write the next file, finish), not on re-reading what you " - "already have. read_file is only for files you did not create.\n" + "file you just wrote. write_file confirms the complete write but may echo only " + "a bounded preview; a 'preview truncated' marker NEVER means the source file " + "was truncated. Spend steps on progress (run it, write the next file, finish). " + "read_file is only for files you did not create.\n" + "- A test command that reports zero discovered tests is a failed test run, even " + "when its exit code is 0. Fix discovery instead of rewriting working source; for " + "Python unittest, put TestCase classes in a test_*.py file.\n" "- To create a missing file, call write_file with its COMPLETE requested " "content. edit_file only changes a file that already exists. Tool names are " "JSON actions, never shell commands.\n" diff --git a/apps/api/app/services/verification.py b/apps/api/app/services/verification.py index 87f06c9..bab3879 100644 --- a/apps/api/app/services/verification.py +++ b/apps/api/app/services/verification.py @@ -25,6 +25,7 @@ from app.tools import CapabilityEnvelope, ToolExecutor, ToolStatus, Workspace from app.tools.guards import make_egress_guard +from app.tools.shell import empty_test_suite_reason def _leading_exit_code(out: str) -> int | None: @@ -176,8 +177,17 @@ def make_result(target: str, passed: bool, evidence: str) -> CheckResult: code = _leading_exit_code(out) # exact code, so "exit code 1" != "exit code 10" exit_ok = code == want_exit stdout_ok = (expect_stdout is None) or (str(expect_stdout) in out) - passed = bool(exit_ok and stdout_ok and tool_result.status is not ToolStatus.BLOCKED) - return make_result(command, passed, out[:300]) + empty_suite = empty_test_suite_reason(command, out) + passed = bool( + exit_ok + and stdout_ok + and empty_suite is None + and tool_result.status is not ToolStatus.BLOCKED + ) + evidence = out[:300] + if empty_suite is not None: + evidence = f"{out[:220]}\nrejected: {empty_suite}"[:300] + return make_result(command, passed, evidence) if kind == "file_exists": path = str(check.get("path", "")).strip() diff --git a/apps/api/app/tools/shell.py b/apps/api/app/tools/shell.py index b62f2f1..c7f2cd5 100644 --- a/apps/api/app/tools/shell.py +++ b/apps/api/app/tools/shell.py @@ -11,6 +11,7 @@ import asyncio import contextlib import os +import re import signal from app.tools.base import ToolResult, ToolStatus @@ -31,6 +32,22 @@ _READ_CAP_MULT = 4 # bytes kept before a runaway-output command is killed +def empty_test_suite_reason(command: str, observation: str) -> str | None: + if re.search(r"(?mi)^Ran 0 tests?\b", observation): + return "the test runner executed zero tests" + runner = re.search( + r"(?i)(?:^|[\s;&|])(?:pytest|py\.test|vitest|jest|mocha)(?:[\s;&|]|$)" + r"|(?:npm|pnpm|yarn|bun)\s+(?:run\s+)?test(?:[\s;&|]|$)", + command, + ) + if runner and re.search( + r"(?i)\b(?:no tests ran|collected 0 items|no tests found|no test files found|0 passing)\b", + observation, + ): + return "the test runner reported an empty test suite" + return None + + def _safe_env() -> dict[str, str]: env = {k: os.environ[k] for k in _SAFE_ENV_KEYS if k in os.environ} env.setdefault("PATH", "/usr/local/bin:/usr/bin:/bin") @@ -115,4 +132,14 @@ async def run_command( raw, code = await collect_output( proc, timeout_seconds=timeout_seconds, output_limit=output_limit ) - return format_result(raw, code, timeout_seconds=timeout_seconds, output_limit=output_limit) + result = format_result(raw, code, timeout_seconds=timeout_seconds, output_limit=output_limit) + empty_suite = empty_test_suite_reason(command, result.observation) + if empty_suite is not None: + return ToolResult( + result.observation + + "\n[INVALID TEST RUN: zero tests were executed. This is not evidence that the " + "source file was truncated. For unittest discovery, put TestCase classes in a " + "test_*.py file; for other runners, add a discoverable test file.]", + ToolStatus.ERROR, + ) + return result diff --git a/apps/api/app/tools/workspace.py b/apps/api/app/tools/workspace.py index 39fc534..f024942 100644 --- a/apps/api/app/tools/workspace.py +++ b/apps/api/app/tools/workspace.py @@ -32,7 +32,9 @@ def _preview(content: str, *, max_lines: int = 20, max_chars: int = 1000) -> str """A bounded echo of written content — enough to confirm the write.""" snippet = "\n".join(content.splitlines()[:max_lines]) truncated = len(snippet) > max_chars or snippet != content.rstrip("\n") - return snippet[:max_chars] + ("\n… (truncated)" if truncated else "") + return snippet[:max_chars] + ( + "\n… [preview truncated; the write completed with the full content]" if truncated else "" + ) class Workspace: @@ -95,7 +97,9 @@ def read(self, relative: str, *, limit: int = 6000) -> str: raise ToolError(f"No such file: {relative}") text = target.read_text(encoding="utf-8", errors="replace") if len(text) > limit: - return text[:limit] + f"\n... [truncated, {len(text)} chars total]" + return text[:limit] + ( + f"\n... [preview truncated; the file is unchanged and has {len(text)} chars total]" + ) return text def list_files(self, *, max_entries: int = 500) -> list[tuple[str, int]]: diff --git a/apps/api/tests/test_agent_react.py b/apps/api/tests/test_agent_react.py index 8de381d..b7f33e6 100644 --- a/apps/api/tests/test_agent_react.py +++ b/apps/api/tests/test_agent_react.py @@ -765,7 +765,7 @@ async def test_strict_contract_runs_required_check_and_maps_every_criterion( assert steps[-1].thought.startswith("[Loop]") -async def test_complete_contract_overrides_failing_supplementary_agent_check( +async def test_run_command_auto_finishes_complete_contract_without_redundant_agent_check( session: AsyncSession, monkeypatch: pytest.MonkeyPatch ) -> None: monkeypatch.setattr(settings, "agent_sandbox", "off") @@ -816,7 +816,11 @@ async def test_complete_contract_overrides_failing_supplementary_agent_check( assert task.stop_reason == StopReason.GOAL_ACHIEVED.value receipt = json.loads(Workspace(Path(task.workspace_path)).read("receipt.json")) assert receipt["checks_passed"] is True - assert receipt["checks"][1]["gating"] is False + assert len(receipt["checks"]) == 1 + assert receipt["checks"][0]["source"] == "contract" + steps = await StepRepository(session).list_for_task(task.id) + assert [step.tool for step in steps] == ["write_file", "run_command", "finish"] + assert steps[-1].thought.startswith("[Loop]") replay = await TaskService(TaskRepository(session), StepRepository(session)).replay_receipt( task.id ) @@ -1654,6 +1658,25 @@ def test_evidence_tool_history_keeps_useful_context_but_stays_bounded() -> None: assert "truncated" in research and "truncated" in local +async def test_history_view_preserves_latest_local_failure_diagnostic( + session: AsyncSession, +) -> None: + diagnostic = "x" * 900 + "\nAssertionError: 4 != 3" + svc = _service(session, ScriptedLLM([])) + svc._history = [ + HistoryEntry( + 1, + "run tests", + "run_command", + {"command": "python3 -m unittest discover -v"}, + diagnostic, + ToolStatus.ERROR, + ) + ] + + assert "AssertionError: 4 != 3" in svc._history_view() + + def test_progress_guard_caps_exploration_without_workspace_progress() -> None: guard = ProgressGuard([]) for index in range(settings.agent_exploration_branch_cap): diff --git a/apps/api/tests/test_completion.py b/apps/api/tests/test_completion.py index 211e5dc..5fc07ed 100644 --- a/apps/api/tests/test_completion.py +++ b/apps/api/tests/test_completion.py @@ -5,6 +5,7 @@ from app.services.completion import ( attach_baseline, completion_gates_pass, + declared_contract_coverage_complete, discover_project_checks, mark_supplementary_agent_checks, merge_completion_checks, @@ -63,6 +64,35 @@ def test_distinct_assertions_on_the_same_file_are_not_deduplicated() -> None: assert [check["text"] for check in checks] == ["Summary", "Risks"] +def test_declared_contract_coverage_supersedes_redundant_agent_checks() -> None: + complete = merge_completion_checks( + [ + { + "kind": "command", + "command": "python3 check.py", + "source": "contract", + } + ], + [], + criterion_count=2, + ) + partial = merge_completion_checks( + [ + { + "kind": "file_exists", + "path": "result.json", + "source": "contract", + "criterion_ids": ["criterion-001"], + } + ], + [], + criterion_count=2, + ) + + assert declared_contract_coverage_complete(complete, 2) is True + assert declared_contract_coverage_complete(partial, 2) is False + + def test_only_new_system_regressions_block_completion() -> None: results = [ CheckResult("command", "lint", False, "still failing", check_id="lint", source="system"), diff --git a/apps/api/tests/test_evaluation.py b/apps/api/tests/test_evaluation.py index e686857..e623b34 100644 --- a/apps/api/tests/test_evaluation.py +++ b/apps/api/tests/test_evaluation.py @@ -2,6 +2,7 @@ import json import re +import subprocess from pathlib import Path from scripts.evaluate_verified_completion import _build_task_payload @@ -75,6 +76,20 @@ def test_aggregate_reports_cost_and_duration() -> None: assert summary["average_duration_seconds"] == 2 +def test_aggregate_rounds_report_metrics_without_float_noise() -> None: + summary = aggregate_verified_completion( + [ + {"solved": True, "duration_seconds": 0.1}, + {"solved": False, "duration_seconds": 0.2}, + {"solved": False, "duration_seconds": 0.3}, + ] + ) + + assert summary["solve_rate"] == 0.3333 + assert summary["total_duration_seconds"] == 0.6 + assert summary["average_duration_seconds"] == 0.2 + + def test_expected_artifacts_are_part_of_the_published_task_contract() -> None: payload = _build_task_payload( { @@ -98,3 +113,35 @@ def test_eval_commands_use_portable_standard_library_python() -> None: assert all(not re.search(r"(^|[;&|]\s*)python(?:\s|$)", command) for command in commands) assert all("pytest" not in command for command in commands) + + +def test_semantic_html_oracle_accepts_valid_thead_and_rejects_wrong_rows(tmp_path: Path) -> None: + cases = json.loads(EVAL_MANIFEST.read_text())["cases"] + case = next(case for case in cases if case["id"] == "semantic-html-report") + command = case["verification_commands"][0] + valid = """ +Scores + + +
Student scores
NameScore
Ada98
Linus91
Grace95
""" + (tmp_path / "report.html").write_text(valid) + + accepted = subprocess.run(command, cwd=tmp_path, shell=True, check=False) + (tmp_path / "report.html").write_text(valid.replace("Grace", "Wrong")) + rejected = subprocess.run(command, cwd=tmp_path, shell=True, check=False) + (tmp_path / "report.html").write_text(valid.replace("Student scores", "")) + empty_caption = subprocess.run(command, cwd=tmp_path, shell=True, check=False) + + assert accepted.returncode == 0 + assert rejected.returncode != 0 + assert empty_caption.returncode != 0 + + +def test_library_cases_require_discoverable_tests_and_external_behavior_oracles() -> None: + cases = {case["id"]: case for case in json.loads(EVAL_MANIFEST.read_text())["cases"]} + + assert "test_arithmetic.py" in cases["python-library"]["expected_files"] + assert "test_string_utils.py" in cases["tested-string-utils"]["expected_files"] + for case_id in ("python-library", "tested-string-utils"): + command = cases[case_id]["verification_commands"][0] + assert command.startswith("python3 -m unittest discover -v && python3 -c") diff --git a/apps/api/tests/test_tools.py b/apps/api/tests/test_tools.py index 8ed281e..e0a0303 100644 --- a/apps/api/tests/test_tools.py +++ b/apps/api/tests/test_tools.py @@ -31,6 +31,17 @@ def test_workspace_write_read_roundtrip(tmp_path: Path) -> None: assert ("notes/a.txt", 5) in ws.list_files() +def test_workspace_preview_distinguishes_display_truncation_from_file_content( + tmp_path: Path, +) -> None: + ws = Workspace(tmp_path / "ws") + observation = ws.write("long.txt", "line\n" * 30) + + assert "preview truncated" in observation + assert "write completed with the full content" in observation + assert ws.read("long.txt") == "line\n" * 30 + + def test_workspace_hides_and_blocks_git_internals(tmp_path: Path) -> None: ws = Workspace(tmp_path / "ws") (ws.root / ".git" / "objects").mkdir(parents=True) @@ -364,6 +375,17 @@ async def test_run_command_caps_runaway_output(tmp_path) -> None: assert len(res.observation) < 20000 # bounded, not ~500KB +async def test_run_command_marks_empty_test_suite_as_error_with_actionable_guidance( + tmp_path: Path, +) -> None: + res = await run_command("python3 -m unittest discover -v", tmp_path) + + assert res.status is ToolStatus.ERROR + assert "INVALID TEST RUN" in res.observation + assert "test_*.py" in res.observation + assert "not evidence that the source file was truncated" in res.observation + + async def test_egress_guard_blocks_network_via_script_file(tmp_path) -> None: # Default-deny egress must also block a script that reaches the network, not # just network commands — else `python fetch.py` slips past on the inline path. diff --git a/apps/api/tests/test_verification.py b/apps/api/tests/test_verification.py index 1b68e05..8add5a2 100644 --- a/apps/api/tests/test_verification.py +++ b/apps/api/tests/test_verification.py @@ -27,6 +27,16 @@ async def test_command_check_fails_on_wrong_output(tmp_path: Path) -> None: assert results[0].passed is False +async def test_command_check_rejects_zero_discovered_tests(tmp_path: Path) -> None: + ws = Workspace(tmp_path / "ws") + results = await run_checks( + [{"kind": "command", "command": "python3 -m unittest discover -v"}], ws + ) + + assert results[0].passed is False + assert "zero tests" in results[0].evidence + + async def test_file_checks(tmp_path: Path) -> None: ws = Workspace(tmp_path / "ws") ws.write("data.csv", "a,b\n1,2\n") diff --git a/evals/README.md b/evals/README.md index 7f99a3c..dc8e3ad 100644 --- a/evals/README.md +++ b/evals/README.md @@ -25,7 +25,9 @@ not a false acceptance. manifest runs on macOS, Linux, and the sandbox image without host-only aliases. Expected artifacts are appended to the task's published acceptance criteria, enforced through the API's `required_artifacts` contract, and checked again by - the scorer; there are no hidden file gates. + the scorer; there are no hidden file gates. Cases that ask the agent to write its + own tests also include visible external behavior assertions, and an empty test + suite is a deterministic failure even when the runner exits zero. ## Zero-cost smoke @@ -71,7 +73,7 @@ provider cost. [`results/deepseek-chat-v0.1.0.json`](./results/deepseek-chat-v0.1.0.json) is one clean run of all 12 cases with DeepSeek `deepseek-chat`: 12 solved, zero false -acceptances, 39 steps, 64,447 provider-reported tokens, and 94.954 seconds. Every +acceptances, 30 steps, 42,403 provider-reported tokens, and 65.795 seconds. Every case passed execution verification, contract coverage, artifact presence, Receipt integrity, and replay. diff --git a/evals/results/deepseek-chat-v0.1.0.json b/evals/results/deepseek-chat-v0.1.0.json index bb91387..27fcc69 100644 --- a/evals/results/deepseek-chat-v0.1.0.json +++ b/evals/results/deepseek-chat-v0.1.0.json @@ -1,32 +1,32 @@ { "label": "deepseek-chat-v0.1.0", "manifest": "evals/verified-completion.json", - "manifest_sha256": "a7cb25e691291b248047ff2e4f5dd4e37fda105aa864c915f63eed7efbe1c89e", + "manifest_sha256": "ac37b8e59f2e52ebbceacb5d4e388cf0836e7f3a5efb7fde948d0362f64a6509", "results": [ { "accepted": true, "category": "code-and-tests", "checks_passed": true, "contract_covered": true, - "duration_seconds": 19.415, + "duration_seconds": 11.892, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "python-library", "integrity_valid": true, "isolation": "inline", - "ledger_head": "16075b14941065204095e69b1889b7c420b275358f9f96af96bd62c04d036b37", + "ledger_head": "ac0565c3dac1109751fbd3e37d00e63ec9160748ff1c99bb5e1ea62c1a86185f", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "fb758034e99e3a5d802bb50d38f8d5f45945b4c62d2f33226fb376448f4390a3", + "receipt_hash": "53e533f608fced0c365665b6e808b0d078898ea297c1d994eed03ff51562e838", "replay_passed": true, "solved": true, "status": "completed", - "steps_used": 5, - "task_id": "a7487645-17f8-4621-afa5-e1fe55adb943", - "tokens_used": 10450 + "steps_used": 3, + "task_id": "8968c015-6c41-442f-a31c-f55304078d55", + "tokens_used": 5855 }, { "accepted": true, @@ -40,283 +40,283 @@ "id": "structured-output", "integrity_valid": true, "isolation": "inline", - "ledger_head": "88f2a6dd412b94319559fd36ca489e988a72d00d71939709d697304da49ffe34", + "ledger_head": "74ad110ce9745239b8bfe85b71089097306594e082d48fbc151855d51d99c802", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "504d162519b260d70d56c8091726f0e78d6c96292d8a4a134abed95c09f454fd", + "receipt_hash": "1f106e0e9253d0feee9b78513af344aa98466ed3f86007c9c2dae962a8303298", "replay_passed": true, "solved": true, "status": "completed", "steps_used": 2, - "task_id": "d21e155c-c035-47e3-89b0-752594704ff3", - "tokens_used": 1895 + "task_id": "9b40a7f4-0981-4c3c-8074-dbc048d182ef", + "tokens_used": 1952 }, { "accepted": true, "category": "document", "checks_passed": true, "contract_covered": true, - "duration_seconds": 3.596, + "duration_seconds": 6.202, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "document-constraints", "integrity_valid": true, "isolation": "inline", - "ledger_head": "3f57a69e6974ff50a9a0a8445860c68a81f891f0b4f1c22eeddd3fd705b136a7", + "ledger_head": "e74ffef4322f23a3e4a1038d58ba29fe2bb2d8318c5d5d07d98961bc594ed13a", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "f339bb4c261d529c2b169636153dc16b4472eee6059ac690827e0871a275017f", + "receipt_hash": "be4fd0349b8f69e8440fa9965e6751179194434f00699f06c7cb85881b0fbdfa", "replay_passed": true, "solved": true, "status": "completed", "steps_used": 2, - "task_id": "bf87764c-db42-4bad-b993-df09fa7f9027", - "tokens_used": 2133 + "task_id": "a9d14206-7e01-4bfe-aaf7-58f189b45da2", + "tokens_used": 2393 }, { "accepted": true, "category": "data-transform", "checks_passed": true, "contract_covered": true, - "duration_seconds": 11.322, + "duration_seconds": 5.698, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "csv-summary", "integrity_valid": true, "isolation": "inline", - "ledger_head": "5a1451edc856ac8b6e1f30d4943fffb0596126c1ba7b3fb6191912ba6f047be2", + "ledger_head": "7da83ac7391a8fea48f2b98af7c6374a7703c5995cf13a8115516bb832e171d9", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "7b3d9ef5068c5b3794ad5c58f232cc1158043d108b9aa717898348f002d1d738", + "receipt_hash": "4ef7f9339a9e438cbc5d5642146515091f7ffb0643d93b142d11c76e6d165daf", "replay_passed": true, "solved": true, "status": "completed", - "steps_used": 5, - "task_id": "df579389-862e-472a-bb25-c225f2b9965d", - "tokens_used": 9985 + "steps_used": 4, + "task_id": "cbc342c6-e4fa-4bfb-81f0-8c037b02713b", + "tokens_used": 5742 }, { "accepted": true, "category": "cli", "checks_passed": true, "contract_covered": true, - "duration_seconds": 4.177, + "duration_seconds": 4.694, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "slug-cli", "integrity_valid": true, "isolation": "inline", - "ledger_head": "1977c6d0a3df05f094a618d66fcc4261aaef55f49ec7ecc7b02d5b62ca6cdfb3", + "ledger_head": "411ef6f480a1298949d79e9f6f14b95b0fd734268df0c609253ae7ba34cd0ab2", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "9663556c5df1169cdf13d5a85ca68b8c8c9c2c3f157effd2dad7d40341654fd4", + "receipt_hash": "ec9e05cb9cd210ce85cdba785b5b2a4bd5b50fc72523089275a793812cf5bca4", "replay_passed": true, "solved": true, "status": "completed", "steps_used": 2, - "task_id": "5af5fe22-485d-48cf-b162-652e55348980", - "tokens_used": 2317 + "task_id": "881fabc1-946b-4ed9-8f5d-60f637bebf12", + "tokens_used": 2506 }, { "accepted": true, "category": "library", "checks_passed": true, "contract_covered": true, - "duration_seconds": 3.619, + "duration_seconds": 4.143, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "deep-config-merge", "integrity_valid": true, "isolation": "inline", - "ledger_head": "e7cbebb43d7f47045b3e8c51907497b95f96bbf38a946772a433d3412871f9bd", + "ledger_head": "2129048391b4b7566557c6df405b72ef2a81c416d6b0b990d05d58574b2caba6", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "b9aab38f4875d2a42452ee9da9d6f301c2c3e4e6da7ab6975c5e2a85ba7af4c7", + "receipt_hash": "f6e0c44686b3dd0c913fa9d7bbcbbf3c6678981d1c3f7331d61ee731b45fd5b2", "replay_passed": true, "solved": true, "status": "completed", "steps_used": 2, - "task_id": "dbea7a05-8d0e-4dbe-b465-79ebd463b830", - "tokens_used": 2352 + "task_id": "0978e7c2-3d21-4e9b-bd12-4f916f77a436", + "tokens_used": 2500 }, { "accepted": true, "category": "state-machine", "checks_passed": true, "contract_covered": true, - "duration_seconds": 3.118, + "duration_seconds": 4.147, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "workflow-state-machine", "integrity_valid": true, "isolation": "inline", - "ledger_head": "850255c4ed48a34d962df27e98874de10ad2d154aeca0df8fecf6eaedfdc6c2a", + "ledger_head": "ff7bbaf1ea4ac202f1a35791fa8f2f235c7c072b5c6c912e9a3557c370960c11", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "51808b9026a11714fd61087442d8e82beaddd14d7bf37cf99526349e14e3c82a", + "receipt_hash": "f22827d4d7d14d4d48e791631c10641239d231c9c18e5641e80e2900bb5cad27", "replay_passed": true, "solved": true, "status": "completed", "steps_used": 2, - "task_id": "507ee1ec-5a2b-4552-8f1d-f76db76c2c88", - "tokens_used": 2142 + "task_id": "ad6d6ccb-c251-41f0-9ff2-aff0892fcfa3", + "tokens_used": 2297 }, { "accepted": true, "category": "frontend-artifact", "checks_passed": true, "contract_covered": true, - "duration_seconds": 18.381, + "duration_seconds": 3.635, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "semantic-html-report", "integrity_valid": true, "isolation": "inline", - "ledger_head": "86e1f67b3260cb95d4276f05905d37b639a551415390464af7f8a0f9e5eb8380", + "ledger_head": "6a14b7858c68606f8272ade5d8e8ce76307be9cfb60afe5e634983ea775ea85b", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "c464d8ec25f09f8b378bbb600ebf03ed17800d28efe07e534d6fa2ba4c34fa0a", + "receipt_hash": "0b3ca372506e9cfaac6b3a41765d13a0078c15ee8097c0451b7b718915d673b9", "replay_passed": true, "solved": true, "status": "completed", - "steps_used": 6, - "task_id": "5fda10e7-bb00-422e-a4fc-9efdda51b4ab", - "tokens_used": 12631 + "steps_used": 2, + "task_id": "ff8e5ef4-ad56-4b26-a566-a19f51800543", + "tokens_used": 2853 }, { "accepted": true, "category": "data-pipeline", "checks_passed": true, "contract_covered": true, - "duration_seconds": 11.864, + "duration_seconds": 7.23, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "jsonl-aggregation", "integrity_valid": true, "isolation": "inline", - "ledger_head": "28628b4822873ffc49617efc706c55665b38f95492a02977c34af91976f7feb1", + "ledger_head": "f4e660c33e9c7f82ab78736bcb8b08c055de09d1fac6bb946d2cc522d8facba8", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "8779bdadf976b84a6eece94a16ef26ea9fcbea39f290ed530282fa19c896160f", + "receipt_hash": "1808a22b843551d1ad3d583a7dbc4749468e93f7bb8a383116a73d533cb2712a", "replay_passed": true, "solved": true, "status": "completed", - "steps_used": 5, - "task_id": "9a9f8865-dda7-4657-9eef-070e78ff3ab0", - "tokens_used": 10544 + "steps_used": 4, + "task_id": "5d763490-dbaa-4551-bb70-cf506f155e9e", + "tokens_used": 6191 }, { "accepted": true, "category": "reliability", "checks_passed": true, "contract_covered": true, - "duration_seconds": 3.095, + "duration_seconds": 3.122, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "bounded-retry", "integrity_valid": true, "isolation": "inline", - "ledger_head": "4f190d7949791c4863cf8324362ec684c06c7ac83084a6c06b3d728c4751c885", + "ledger_head": "ed3a96b05355285bf2fc6ceccac550f95ad460a72635eac85f656674b98b3858", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "698f950d0a6f3e6d59af5501c7a6f6d7572f9c0f8bff5d85426a806dd7a28e0e", + "receipt_hash": "605b65828e29dfb6bd36c54dce763e0526eee6c7d1db45a10b8a08fcf7d7acf7", "replay_passed": true, "solved": true, "status": "completed", "steps_used": 2, - "task_id": "4c6adbab-c3dc-404e-b203-d7600db1b990", - "tokens_used": 2233 + "task_id": "3ec3bc8e-8a80-49bc-a22a-76599ededec0", + "tokens_used": 2298 }, { "accepted": true, "category": "algorithm", "checks_passed": true, "contract_covered": true, - "duration_seconds": 3.584, + "duration_seconds": 4.154, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "inverted-index", "integrity_valid": true, "isolation": "inline", - "ledger_head": "37fdc1906f218a1866c4fc91ae959c4f04ddd422189c4b3445c477b7c681a46a", + "ledger_head": "dc9ce7a4383d8163d9aa8049b4ee71a1040f16a7a2afc12ec7d5985019da8950", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "7bc1854a7440d7ad94d4e698edea2126c8a618ad844f25dfee9deb3d17ade1de", + "receipt_hash": "0dada1714dcfd70d6d321409fa8ebf9def4a893352a034f86bc76a8f75e5869a", "replay_passed": true, "solved": true, "status": "completed", "steps_used": 2, - "task_id": "d537ad1f-078e-4ed3-b0ce-467911956a20", - "tokens_used": 2192 + "task_id": "06e340a1-5f68-4a04-a0fe-b9afbbd7c770", + "tokens_used": 2376 }, { "accepted": true, "category": "code-and-tests", "checks_passed": true, "contract_covered": true, - "duration_seconds": 10.165, + "duration_seconds": 8.26, "execution_verified": true, "expected_files_present": true, "false_acceptance": false, "id": "tested-string-utils", "integrity_valid": true, "isolation": "inline", - "ledger_head": "b7a99f60041ee5ff60ee89bac74a55957bc816aa9ea08b54273163400040c70a", + "ledger_head": "6f545b59c1a06082a0af71b9806351b47b99a193ef5392fc331cbe4c5817f4b4", "model": { "model": "deepseek-chat", "provider": "deepseek" }, - "receipt_hash": "bbb553b60cc085e1d574e139270af9caa31d8e5c98c09878b1eeb26464200c2a", + "receipt_hash": "7693410e75e47aae79635af65fd8b8da16e137d72de5db74ce2b22d53958ea06", "replay_passed": true, "solved": true, "status": "completed", - "steps_used": 4, - "task_id": "58ee88e1-5f4b-4dc8-9b94-eaa528378c9a", - "tokens_used": 5573 + "steps_used": 3, + "task_id": "e8aa7221-bc6b-422f-9ff0-7ce629da367b", + "tokens_used": 5440 } ], - "run_at": "2026-07-16T01:18:32.951480+00:00", + "run_at": "2026-07-16T03:19:30.253283+00:00", "schema": "loop.verified-completion-eval/v1", "summary": { - "average_duration_seconds": 7.9128333333333325, - "average_steps": 3.25, - "average_tokens": 5370.583333333333, + "average_duration_seconds": 5.483, + "average_steps": 2.5, + "average_tokens": 3533.583, "cases": 12, "false_acceptance_rate": 0.0, "false_acceptances": 0, "solve_rate": 1.0, "solved": 12, - "total_duration_seconds": 94.954, - "total_steps": 39, - "total_tokens": 64447 + "total_duration_seconds": 65.795, + "total_steps": 30, + "total_tokens": 42403 } } diff --git a/evals/verified-completion.json b/evals/verified-completion.json index ce46b06..344784d 100644 --- a/evals/verified-completion.json +++ b/evals/verified-completion.json @@ -10,8 +10,10 @@ "Division by zero raises ZeroDivisionError.", "The automated test suite passes." ], - "verification_commands": ["python3 -m unittest discover -v"], - "expected_files": ["arithmetic.py"] + "verification_commands": [ + "python3 -m unittest discover -v && python3 -c \"from arithmetic import add,subtract,multiply,divide; assert add(2,3)==5 and subtract(7,4)==3 and multiply(-2,5)==-10 and divide(5,2)==2.5; exec('try:\\n divide(1,0)\\nexcept ZeroDivisionError:\\n pass\\nelse:\\n raise AssertionError')\"" + ], + "expected_files": ["arithmetic.py", "test_arithmetic.py"] }, { "id": "structured-output", @@ -104,7 +106,7 @@ "Every requested name and score is present." ], "verification_commands": [ - "python3 -c \"from html.parser import HTMLParser; from pathlib import Path; s=Path('report.html').read_text(); assert all(x in s for x in ['