diff --git a/containers/simulation_heuristics_ant_v1/Dockerfile b/containers/simulation_heuristics_ant_v1/Dockerfile deleted file mode 100644 index 286c43e..0000000 --- a/containers/simulation_heuristics_ant_v1/Dockerfile +++ /dev/null @@ -1,27 +0,0 @@ -FROM ghcr.io/openhands/agent-server:latest-python - -USER root - -RUN /usr/local/venv/bin/python -m pip install --no-cache-dir \ - "openhands-sdk==1.34.0" \ - "openhands-tools==1.34.0" \ - "gymnasium[mujoco]>=1.2.2,<2" \ - "numpy>=1.26,<3" \ - "pyyaml>=6,<7" - -RUN npm install --global --force @agentclientprotocol/claude-agent-acp@0.58.1 - -RUN rm -f /usr/local/bin/python /usr/local/bin/pip \ - && printf '#!/bin/sh\nexec /usr/local/venv/bin/python "$@"\n' \ - > /usr/local/bin/python \ - && printf '#!/bin/sh\nexec /usr/local/venv/bin/pip "$@"\n' \ - > /usr/local/bin/pip \ - && chmod +x /usr/local/bin/python /usr/local/bin/pip - -COPY scripts/run_openhands_agent.py /opt/genesisbench/run_openhands_agent.py - -ENV PATH="/usr/local/venv/bin:${PATH}" - -WORKDIR /workspace - -ENTRYPOINT ["/usr/local/venv/bin/python", "/opt/genesisbench/run_openhands_agent.py"] diff --git a/docs/simulation-heuristics-ant-v1-task-design.md b/docs/simulation-heuristics-ant-v1-task-design.md index 9a4cb80..4516a87 100644 --- a/docs/simulation-heuristics-ant-v1-task-design.md +++ b/docs/simulation-heuristics-ant-v1-task-design.md @@ -199,8 +199,8 @@ GENESISBENCH_RUN_SLOW_ANT_MPC=1 \ - The agent container receives the public task but not `verifier/`. - The agent receives only a copied model path for optional MPC, never the live scored environment. -- Credentials are supplied through a temporary mode-`0600` file and removed - after agent startup. +- Provider credentials are scoped to the selected model route and are not + copied into the public task workspace or committed artifacts. - Final scores come from the independently imported final policy. - Invalid actions receive a fixed failure result instead of crashing scoring. - Public packaged score files contain only relative paths. @@ -213,7 +213,7 @@ Simulation Heuristics Ant v1 demonstrates the full GenesisBench task lifecycle: ```text task scaffold → public workspace preparation -→ autonomous OpenHands run +→ autonomous BenchFlow/OpenCode run → clean final evaluation → packaged submission → audited leaderboard diff --git a/experiments/simulation_heuristics_ant_v1/README.md b/experiments/simulation_heuristics_ant_v1/README.md deleted file mode 100644 index 8a42b45..0000000 --- a/experiments/simulation_heuristics_ant_v1/README.md +++ /dev/null @@ -1,117 +0,0 @@ -# Legacy Simulation Heuristics Ant v1 OpenHands sweep - -This directory documents the historical Ant-only sweep. It is retained so the -published Ant leaderboard can be reproduced and audited. New experiments must -use the OpenCode article-suite runner documented in -`experiments/article_suite/README.md`. - -This experiment runs the same GenesisBench -`simulation_heuristics_ant_v1` policy-improvement task with four model -configurations through OpenHands. - -## Models - -| ID | Provider route | Reasoning | -| --- | --- | --- | -| `gpt-5.6-sol` | Azure `gpt-5.6-sol` | `xhigh` | -| `gpt-5.5` | Azure `gpt-5.5` | `xhigh` | -| `claude-opus-4.8` | OpenHands `ACPAgent` + Claude Agent ACP | `max` | -| `gpt-5.4-mini` | Azure `gpt-5.4-mini` | `xhigh` | - -The three GPT models use the standard OpenHands SDK agent loop. Opus 4.8 uses -OpenHands' supported `ACPAgent` integration because the available credential is -a Claude subscription credential rather than an Anthropic API key. Both paths -run inside an OpenHands `Conversation` and receive isolated copies of the same -task. - -## Run one model - -```bash -uv sync --extra dev -uv run python scripts/run_simulation_heuristics_ant_v1_experiment.py \ - --model gpt-5.6-sol \ - --minutes 30 \ - --max-iterations 500 -``` - -The authoritative runtime is Docker. Build it once: - -```bash -sh scripts/build_simulation_heuristics_ant_v1_runner_image.sh -``` - -`--runtime local` is available only for harness development. Local runs are -not leaderboard-valid because a local terminal could traverse outside the task -directory. - -Copy the credential template and fill only the providers you plan to run: - -```bash -cp .env.example .env -``` - -Alternatively, pass another file: - -```bash -uv run python scripts/run_simulation_heuristics_ant_v1_experiment.py \ - --model gpt-5.6-sol \ - --env-file /secure/path/provider.env -``` - -Credentials are written only to a mode-`0600` temporary run file, removed -after the agent exits, and never copied into the task workspace. - -Docker defaults to the host architecture. Override it when needed: - -```bash -GENESISBENCH_DOCKER_PLATFORM=linux/amd64 \ - sh scripts/build_simulation_heuristics_ant_v1_runner_image.sh -``` - -## Run artifacts - -Each run stores: - -```text -leaderboard/runs/// - workspace/ # isolated public task and submitted final policy - events.jsonl # OpenHands event stream - conversation/ # OpenHands persisted conversation - model_config.json # non-secret model configuration - agent_summary.json # usage and timing when the agent exits normally - run_metadata.json # budget, timeout, process, and scoring metadata - score.json # authoritative hidden evaluation -``` - -The model never receives `tasks/simulation_heuristics_ant_v1/verifier/`. Scoring happens after the -agent exits, in the GenesisBench repository's clean evaluator. - -`leaderboard/runs/` is gitignored. Durable public artifacts are the sanitized -policies, score files, metadata, chart, and report under `leaderboard/`. - -## Rebuild the leaderboard - -```bash -uv run python scripts/build_leaderboard.py -``` - -This selects the latest completed score for each model and writes: - -```text -leaderboard/simulation_heuristics_ant_v1.json -leaderboard/README.md -``` - -## Fairness notes - -- All agents receive the same task files and starter policy. -- Authoritative runs execute in a container that mounts the public task but not - the hidden verifier. -- All agents receive the same wall-clock budget. -- GPT reasoning effort is `xhigh`, the highest effort exposed by the installed - OpenHands SDK for those routes. -- Opus effort is `max`, configured in its isolated Claude settings. -- Public development seeds are visible. -- Final nominal seeds and dynamics configurations are not copied to the agent. -- The leaderboard score comes only from `evaluate_hidden.py`, never from the - agent's self-reported development results. diff --git a/experiments/simulation_heuristics_ant_v1/models.toml b/experiments/simulation_heuristics_ant_v1/models.toml deleted file mode 100644 index 4dc8ebc..0000000 --- a/experiments/simulation_heuristics_ant_v1/models.toml +++ /dev/null @@ -1,29 +0,0 @@ -version = "1.0" - -[[models]] -id = "gpt-5.6-sol" -display_name = "GPT-5.6 Sol" -provider = "azure" -model = "azure/gpt-5.6-sol" -reasoning_effort = "xhigh" - -[[models]] -id = "gpt-5.5" -display_name = "GPT-5.5" -provider = "azure" -model = "azure/gpt-5.5" -reasoning_effort = "xhigh" - -[[models]] -id = "claude-opus-4.8" -display_name = "Claude Opus 4.8" -provider = "claude_acp" -model = "claude-opus-4-8" -thinking_effort = "max" - -[[models]] -id = "gpt-5.4-mini" -display_name = "GPT-5.4 Mini" -provider = "azure" -model = "azure/gpt-5.4-mini" -reasoning_effort = "xhigh" diff --git a/leaderboard/README.md b/leaderboard/README.md index 968603f..e71a775 100644 --- a/leaderboard/README.md +++ b/leaderboard/README.md @@ -11,6 +11,10 @@ nine task-specific rankings followed by the final IQM ranking. ![GenesisBench Simulation Heuristics Ant v1 leaderboard](simulation_heuristics_ant_v1_leaderboard.png) +This is an immutable historical OpenHands result. The retired runner is no +longer shipped in the current tree; see [`REPORT.md`](REPORT.md) for provenance +and the Git-history source pointer. + | Rank | Agent model | Harness | Reasoning | Score | Normalized | Nominal | Robust | Fall rate | | ---: | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | | 1 | GPT-5.6 Sol | OpenHands SDK | xhigh | 3417.86 | 150.24 | 3399.54 | 3460.62 | 0.0% | diff --git a/leaderboard/REPORT.md b/leaderboard/REPORT.md index 6837493..1db32f3 100644 --- a/leaderboard/REPORT.md +++ b/leaderboard/REPORT.md @@ -56,7 +56,7 @@ surviving for more steps. - Machine-readable leaderboard: `leaderboard/simulation_heuristics_ant_v1.json` - Packaged policies and scores: `leaderboard/submissions/` - Deterministic verifier: `tasks/simulation_heuristics_ant_v1/verifier/evaluate_hidden.py` -- Run and audit commands: `experiments/simulation_heuristics_ant_v1/README.md` +- Artifact audit: `scripts/audit_simulation_heuristics_ant_v1_leaderboard.py` Every packaged policy was re-evaluated after packaging, and all four scores matched their recorded score exactly. @@ -65,6 +65,11 @@ Raw OpenHands trajectories are intentionally excluded from the source distribution because they are large and can contain provider-specific runtime metadata. Packaged metadata retains a non-secret source run identifier. +The executable OpenHands runner was retired after the nine-task OpenCode suite +became canonical. Its final source snapshot remains available in Git history at +commit `8f05ed4`; the current tree retains only the immutable published +artifacts, verifier, audit, and this methodology report. + ## Limitations - This is one agent run per model, not a multi-trial estimate of model quality. diff --git a/scripts/build_simulation_heuristics_ant_v1_runner_image.sh b/scripts/build_simulation_heuristics_ant_v1_runner_image.sh deleted file mode 100755 index 9799454..0000000 --- a/scripts/build_simulation_heuristics_ant_v1_runner_image.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/sh -set -eu - -cd "$(dirname "$0")/.." - -case "$(uname -m)" in - arm64|aarch64) default_platform="linux/arm64" ;; - *) default_platform="linux/amd64" ;; -esac - -platform="${GENESISBENCH_DOCKER_PLATFORM:-$default_platform}" -image="${GENESISBENCH_DOCKER_IMAGE:-genesisbench-simulation-heuristics-ant-v1-runner:latest}" - -docker build \ - --platform "$platform" \ - --tag "$image" \ - --file containers/simulation_heuristics_ant_v1/Dockerfile \ - . diff --git a/scripts/run_openhands_agent.py b/scripts/run_openhands_agent.py deleted file mode 100644 index e8b6d69..0000000 --- a/scripts/run_openhands_agent.py +++ /dev/null @@ -1,191 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -import os -import shutil -import sys -import time -from pathlib import Path -from typing import Any - -from pydantic import SecretStr - -from openhands.sdk import Agent, Conversation, Event, LLM -from openhands.sdk.agent.acp_agent import ACPAgent -from openhands.sdk.tool import Tool -from openhands.tools.file_editor import FileEditorTool -from openhands.tools.task_tracker import TaskTrackerTool -from openhands.tools.terminal import TerminalTool - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description="Run one OpenHands agent on a prepared Ant task." - ) - parser.add_argument("--workspace", type=Path, required=True) - parser.add_argument("--run-dir", type=Path, required=True) - parser.add_argument("--model-config", type=Path, required=True) - parser.add_argument("--env-json", type=Path, required=True) - parser.add_argument("--max-iterations", type=int, default=500) - return parser.parse_args() - - -def _event_json(event: Event) -> str: - try: - return event.model_dump_json() - except Exception: - return json.dumps( - { - "type": type(event).__name__, - "rendered": str(event), - }, - ensure_ascii=False, - ) - - -def _create_llm(config: dict[str, Any], env: dict[str, str]) -> LLM: - common: dict[str, Any] = { - "model": config["model"], - "usage_id": config["id"], - "timeout": 300, - "num_retries": 5, - "max_output_tokens": 32768, - "reasoning_summary": "detailed", - "caching_prompt": True, - } - provider = config["provider"] - if provider == "azure": - common.update( - { - "api_key": SecretStr(env["AZURE_API_KEY"]), - "base_url": env["AZURE_API_ENDPOINT"], - "api_version": env.get("AZURE_API_VERSION", "preview"), - "reasoning_effort": config.get("reasoning_effort", "xhigh"), - "extended_thinking_budget": None, - } - ) - else: - raise ValueError(f"Unsupported provider: {provider}") - return LLM(**common) - - -def _create_agent( - config: dict[str, Any], - env: dict[str, str], - run_dir: Path, -) -> tuple[Agent | ACPAgent, LLM | None]: - if config["provider"] == "claude_acp": - claude_config_dir = run_dir / "claude_config" - claude_config_dir.mkdir(parents=True, exist_ok=True) - (claude_config_dir / "settings.json").write_text( - json.dumps( - { - "model": config["model"], - "effortLevel": config.get("thinking_effort", "max"), - "permissions": {"defaultMode": "bypassPermissions"}, - }, - indent=2, - ) - + "\n" - ) - os.environ["CLAUDE_CODE_OAUTH_TOKEN"] = env["CLAUDE_CODE_OAUTH_TOKEN"] - os.environ["CLAUDE_CONFIG_DIR"] = str(claude_config_dir) - os.environ["ANTHROPIC_MODEL"] = config["model"] - acp_env = { - "PATH": os.environ.get("PATH", ""), - "HOME": os.environ.get("HOME", ""), - } - acp_binary = shutil.which("claude-agent-acp") - acp_command = ( - [acp_binary] - if acp_binary is not None - else ["npx", "-y", "@agentclientprotocol/claude-agent-acp"] - ) - return ( - ACPAgent( - acp_command=acp_command, - acp_env=acp_env, - acp_model=config["model"], - acp_session_mode="acceptEdits", - acp_prompt_timeout=1800.0, - ), - None, - ) - - llm = _create_llm(config, env) - tools = [ - Tool( - name=TerminalTool.name, - params={"no_change_timeout_seconds": 5}, - ), - Tool(name=FileEditorTool.name), - Tool(name=TaskTrackerTool.name), - ] - return Agent(llm=llm, tools=tools), llm - - -def main() -> None: - args = parse_args() - workspace = args.workspace.resolve() - run_dir = args.run_dir.resolve() - run_dir.mkdir(parents=True, exist_ok=True) - config = json.loads(args.model_config.read_text()) - env = json.loads(args.env_json.read_text()) - args.env_json.unlink(missing_ok=True) - - event_path = run_dir / "events.jsonl" - event_file = event_path.open("a", encoding="utf-8") - - def callback(event: Event) -> None: - event_file.write(_event_json(event) + "\n") - event_file.flush() - - agent, llm = _create_agent(config, env, run_dir) - conversation = Conversation( - agent=agent, - workspace=workspace, - persistence_dir=run_dir / "conversation", - callbacks=[callback], - max_iteration_per_run=args.max_iterations, - visualizer=None, - delete_on_close=False, - tags={ - "benchmark": "simulation_heuristics_ant_v1", - "model": config["id"], - }, - ) - - sys.path.insert(0, str(workspace / "_runtime")) - from genesisbench.task_document import TaskDocument - - prompt = TaskDocument.from_path(workspace / "task.md").instruction - started_at = time.time() - status = "completed" - error: str | None = None - try: - conversation.send_message(prompt) - conversation.run() - except Exception as exception: - status = "error" - error = f"{type(exception).__name__}: {exception}" - raise - finally: - event_file.close() - summary = { - "status": status, - "error": error, - "model": config, - "started_at": started_at, - "finished_at": time.time(), - "elapsed_seconds": time.time() - started_at, - "metrics": llm.metrics.model_dump(mode="json") if llm else None, - } - (run_dir / "agent_summary.json").write_text( - json.dumps(summary, indent=2, sort_keys=True) + "\n" - ) - - -if __name__ == "__main__": - main() diff --git a/scripts/run_simulation_heuristics_ant_v1_experiment.py b/scripts/run_simulation_heuristics_ant_v1_experiment.py deleted file mode 100644 index 80f4e54..0000000 --- a/scripts/run_simulation_heuristics_ant_v1_experiment.py +++ /dev/null @@ -1,376 +0,0 @@ -#!/usr/bin/env python3 -from __future__ import annotations - -import argparse -import json -import os -import platform -import shutil -import signal -import subprocess -import sys -import time -import tomllib -from datetime import UTC, datetime -from pathlib import Path -from typing import Any - - -REPO_ROOT = Path(__file__).resolve().parents[1] -TASK_NAME = "simulation_heuristics_ant_v1" -MODELS_PATH = REPO_ROOT / "experiments" / TASK_NAME / "models.toml" -DOCKER_IMAGE = os.environ.get( - "GENESISBENCH_DOCKER_IMAGE", - "genesisbench-simulation-heuristics-ant-v1-runner:latest", -) - - -def _default_docker_platform() -> str: - configured = os.environ.get("GENESISBENCH_DOCKER_PLATFORM") - if configured: - return configured - machine = platform.machine().lower() - return "linux/arm64" if machine in {"arm64", "aarch64"} else "linux/amd64" - - -def _default_env_path() -> Path: - return Path( - os.environ.get("GENESISBENCH_ENV_FILE", REPO_ROOT / ".env") - ) - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description="Run and score one OpenHands Ant experiment." - ) - parser.add_argument("--model", required=True) - parser.add_argument("--minutes", type=int, default=30) - parser.add_argument("--max-iterations", type=int, default=500) - parser.add_argument("--output-root", type=Path) - parser.add_argument("--env-file", type=Path, default=_default_env_path()) - parser.add_argument( - "--docker-platform", - default=_default_docker_platform(), - ) - parser.add_argument( - "--openhands-python", - type=Path, - default=None, - help="Python from an OpenHands SDK installation for --runtime local.", - ) - parser.add_argument( - "--runtime", - choices=("docker", "local"), - default="docker", - ) - return parser.parse_args() - - -def _read_env(path: Path) -> dict[str, str]: - result: dict[str, str] = {} - for raw_line in path.read_text().splitlines(): - line = raw_line.strip() - if not line or line.startswith("#") or "=" not in line: - continue - key, value = line.split("=", 1) - value = value.strip() - if ( - len(value) >= 2 - and value[0] == value[-1] - and value[0] in {'"', "'"} - ): - value = value[1:-1] - result[key] = value - return result - - -def _provider_environment(path: Path) -> dict[str, str]: - result = {} - if path.is_file(): - result.update(_read_env(path)) - result.update(os.environ) - return result - - -def _require_provider_keys( - model: dict[str, Any], - provider_env: dict[str, str], -) -> None: - required = { - "azure": ("AZURE_API_ENDPOINT", "AZURE_API_KEY"), - "claude_acp": ("CLAUDE_CODE_OAUTH_TOKEN",), - }[model["provider"]] - missing = [key for key in required if not provider_env.get(key)] - if missing: - joined = ", ".join(missing) - raise RuntimeError( - f"Missing credentials for {model['id']}: {joined}. " - "Set environment variables or pass --env-file." - ) - - -def _resolve_openhands_python(explicit: Path | None) -> Path: - if explicit is not None: - path = explicit.expanduser().resolve() - if not path.is_file(): - raise FileNotFoundError(path) - return path - configured = os.environ.get("GENESISBENCH_OPENHANDS_PYTHON") - if configured: - return _resolve_openhands_python(Path(configured)) - executable = shutil.which("openhands") - if executable: - first_line = Path(executable).read_text().splitlines()[0] - if first_line.startswith("#!"): - candidate = Path(first_line[2:]) - if candidate.is_file(): - return candidate - raise RuntimeError( - "Unable to locate an OpenHands Python runtime. Set " - "GENESISBENCH_OPENHANDS_PYTHON or pass --openhands-python." - ) - - -def _load_model(model_id: str) -> dict[str, Any]: - config = tomllib.loads(MODELS_PATH.read_text()) - for model in config["models"]: - if model["id"] == model_id: - return model - choices = ", ".join(model["id"] for model in config["models"]) - raise ValueError(f"Unknown model {model_id!r}; choose one of: {choices}") - - -def _write_timer(workspace: Path, deadline: float) -> None: - timer = workspace / "timer.sh" - timer.write_text( - "#!/bin/sh\n" - "python - <<'PY'\n" - "import time\n" - f"remaining = max(0, int({deadline!r} - time.time()))\n" - "print(f'remaining_seconds={remaining}')\n" - "PY\n" - ) - timer.chmod(0o755) - - -def _terminate( - process: subprocess.Popen[str], - *, - container_name: str | None = None, -) -> None: - try: - os.killpg(process.pid, signal.SIGTERM) - process.wait(timeout=20) - except (ProcessLookupError, subprocess.TimeoutExpired): - try: - os.killpg(process.pid, signal.SIGKILL) - except ProcessLookupError: - pass - if container_name is not None: - subprocess.run( - ["docker", "rm", "--force", container_name], - check=False, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - - -def main() -> None: - args = parse_args() - model = _load_model(args.model) - timestamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ") - output_root = ( - args.output_root.resolve() - if args.output_root - else REPO_ROOT / "leaderboard" / "runs" / timestamp - ) - run_dir = output_root / model["id"] - workspace = run_dir / "workspace" - run_dir.mkdir(parents=True, exist_ok=True) - - subprocess.run( - [ - sys.executable, - str(REPO_ROOT / "scripts" / "prepare_task.py"), - TASK_NAME, - str(workspace), - "--force", - ], - check=True, - ) - deadline = time.time() + args.minutes * 60 - _write_timer(workspace, deadline) - - model_config_path = run_dir / "model_config.json" - model_config_path.write_text( - json.dumps(model, indent=2, sort_keys=True) + "\n" - ) - provider_env = _provider_environment(args.env_file.expanduser()) - _require_provider_keys(model, provider_env) - safe_env = { - key: provider_env[key] - for key in ( - "AZURE_API_ENDPOINT", - "AZURE_API_KEY", - "AZURE_API_VERSION", - "AWS_BEARER_TOKEN_BEDROCK", - "AWS_REGION", - "CLAUDE_CODE_OAUTH_TOKEN", - ) - if key in provider_env - } - env_json_path = run_dir / ".provider_env.json" - env_json_path.write_text(json.dumps(safe_env)) - env_json_path.chmod(0o600) - - child_env = os.environ.copy() - child_env["OPENHANDS_SUPPRESS_BANNER"] = "1" - child_env["PYTHONUNBUFFERED"] = "1" - container_name: str | None = None - if args.runtime == "docker": - image_check = subprocess.run( - ["docker", "image", "inspect", DOCKER_IMAGE], - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - if image_check.returncode != 0: - raise RuntimeError( - "Missing Docker image. Run scripts/build_simulation_heuristics_ant_v1_runner_image.sh" - ) - container_name = ( - "genesisbench-simulation-heuristics-ant-v1-" - + model["id"].replace(".", "-") - + "-" - + timestamp.lower() - ) - command = [ - "docker", - "run", - "--rm", - "--platform", - args.docker_platform, - "--name", - container_name, - "--volume", - f"{workspace}:/workspace", - "--volume", - f"{run_dir}:/artifacts", - "--workdir", - "/workspace", - "--env", - "OPENHANDS_SUPPRESS_BANNER=1", - DOCKER_IMAGE, - "--workspace", - "/workspace", - "--run-dir", - "/artifacts", - "--model-config", - "/artifacts/model_config.json", - "--env-json", - "/artifacts/.provider_env.json", - "--max-iterations", - str(args.max_iterations), - ] - else: - openhands_python = _resolve_openhands_python(args.openhands_python) - child_env["PATH"] = ( - str(Path(sys.executable).parent) - + os.pathsep - + child_env.get("PATH", "") - ) - command = [ - str(openhands_python), - str(REPO_ROOT / "scripts" / "run_openhands_agent.py"), - "--workspace", - str(workspace), - "--run-dir", - str(run_dir), - "--model-config", - str(model_config_path), - "--env-json", - str(env_json_path), - "--max-iterations", - str(args.max_iterations), - ] - started_at = time.time() - timed_out = False - with (run_dir / "agent_stdout.log").open("w") as stdout, ( - run_dir / "agent_stderr.log" - ).open("w") as stderr: - process = subprocess.Popen( - command, - cwd=workspace, - env=child_env, - stdout=stdout, - stderr=stderr, - text=True, - start_new_session=True, - ) - try: - return_code = process.wait(timeout=args.minutes * 60) - except subprocess.TimeoutExpired: - timed_out = True - _terminate(process, container_name=container_name) - return_code = process.returncode - - env_json_path.unlink(missing_ok=True) - policy_path = workspace / "final_policy" / "policy.py" - score_path = run_dir / "score.json" - score_process = subprocess.run( - [ - sys.executable, - str( - REPO_ROOT - / "tasks" - / TASK_NAME - / "verifier" - / "evaluate_hidden.py" - ), - str(policy_path), - "--output", - str(score_path), - ], - cwd=REPO_ROOT, - text=True, - capture_output=True, - ) - (run_dir / "score_stdout.log").write_text(score_process.stdout) - (run_dir / "score_stderr.log").write_text(score_process.stderr) - - metadata = { - "benchmark": TASK_NAME, - "harness": ( - "OpenHands SDK ACPAgent" - if model["provider"] == "claude_acp" - else "OpenHands SDK" - ), - "openhands_python": ( - str(openhands_python) if args.runtime == "local" else None - ), - "runtime": args.runtime, - "docker_image": DOCKER_IMAGE if args.runtime == "docker" else None, - "docker_platform": ( - args.docker_platform if args.runtime == "docker" else None - ), - "model": model, - "budget_minutes": args.minutes, - "max_iterations": args.max_iterations, - "started_at": started_at, - "finished_at": time.time(), - "elapsed_seconds": time.time() - started_at, - "timed_out": timed_out, - "agent_return_code": return_code, - "score_return_code": score_process.returncode, - "score_file": str(score_path), - } - (run_dir / "run_metadata.json").write_text( - json.dumps(metadata, indent=2, sort_keys=True) + "\n" - ) - print(run_dir) - if score_process.returncode != 0: - print(score_process.stderr, file=sys.stderr) - raise SystemExit(score_process.returncode) - - -if __name__ == "__main__": - main() diff --git a/tasks/README.md b/tasks/README.md index 26307e9..6df1eb7 100644 --- a/tasks/README.md +++ b/tasks/README.md @@ -61,6 +61,20 @@ tasks// Task-specific files and assets may be added as needed. +## Container layout + +Each current task owns its agent environment at: + +```text +tasks//environment/Dockerfile +``` + +BenchFlow builds that task-local Dockerfile with the repository root as its +context. There is intentionally no parallel top-level `containers//` +tree. The former `containers/simulation_heuristics_ant_v1` image was an +OpenHands-specific runner for the retired Ant-only sweep, not a task +environment. + ## Public workspace boundary `scripts/prepare_task.py` copies the agent-visible task and deliberately diff --git a/tasks/simulation_heuristics_ant_v1/README.md b/tasks/simulation_heuristics_ant_v1/README.md index 29c1c09..adf1f80 100644 --- a/tasks/simulation_heuristics_ant_v1/README.md +++ b/tasks/simulation_heuristics_ant_v1/README.md @@ -166,8 +166,8 @@ uv run python scripts/prepare_task.py \ --force ``` -The custom OpenHands experiment workspace must not contain `verifier/`, -`oracle/`, or `evidence/`. +The prepared coding-agent workspace must not contain `verifier/`, `oracle/`, +or `evidence/`. ## Files contributors should study diff --git a/tests/test_task_tooling.py b/tests/test_task_tooling.py index eb3a24b..e48adcd 100644 --- a/tests/test_task_tooling.py +++ b/tests/test_task_tooling.py @@ -12,6 +12,14 @@ REPO_ROOT = Path(__file__).resolve().parents[1] TASKS_ROOT = REPO_ROOT / "tasks" RUNTIME_SOURCE = REPO_ROOT / "src" / "genesisbench" +RETIRED_OPENHANDS_RUNNER_PATHS = ( + "containers/simulation_heuristics_ant_v1/Dockerfile", + "experiments/simulation_heuristics_ant_v1/README.md", + "experiments/simulation_heuristics_ant_v1/models.toml", + "scripts/build_simulation_heuristics_ant_v1_runner_image.sh", + "scripts/run_openhands_agent.py", + "scripts/run_simulation_heuristics_ant_v1_experiment.py", +) def test_article_suite_contains_exactly_nine_valid_tasks() -> None: @@ -26,6 +34,14 @@ def test_article_suite_contains_exactly_nine_valid_tasks() -> None: ) == [] +def test_tasks_own_their_containers_and_legacy_runner_is_removed() -> None: + for task_name in ARTICLE_TASKS: + assert (TASKS_ROOT / task_name / "environment" / "Dockerfile").is_file() + + for relative_path in RETIRED_OPENHANDS_RUNNER_PATHS: + assert not (REPO_ROOT / relative_path).exists() + + def test_create_and_prepare_task_scaffold(tmp_path: Path) -> None: tasks_root = tmp_path / "tasks" shutil.copytree(TASKS_ROOT / "_template", tasks_root / "_template") diff --git a/website/README.md b/website/README.md index 764e2e4..dbf3ec8 100644 --- a/website/README.md +++ b/website/README.md @@ -43,6 +43,10 @@ the official ranking metric. The repo ships `vercel.json` (serve `website/` as-is, no build; deploys are skipped unless `website/` changed). One-time setup: +The `ignoreCommand` compares `website/` and `vercel.json`. Repository-only +maintenance outside those paths is intentionally skipped and does not change +the production site. + 1. **Vercel** (benchflow team) → *Add New Project* → import `benchflow-ai/GenesisBench` → settings are read from `vercel.json` → Deploy. From then on every push to `main` that touches `website/` auto-deploys to production, and every PR gets a preview URL.