diff --git a/containers/benchmarks/appworld/AUDIT.md b/containers/benchmarks/appworld/AUDIT.md new file mode 100644 index 00000000..ca48c246 --- /dev/null +++ b/containers/benchmarks/appworld/AUDIT.md @@ -0,0 +1,128 @@ +--- +benchmark: appworld +host: local Colima (amd64 via Rosetta, no QEMU) +commit: d9ef64a9 +--- +# Audit — appworld + +`✓` verified (a check passed) · `◐` partial (holds in one surface, open in another) · `✗` failing · `?` unchecked · `n/a` not applicable + +## Validity — is the score real? + +| Check | Status | Evidence | +|-------|:------:|----------| +| building | ✓ | `docker build --platform linux/amd64` succeeds, no-cache, 28s of `RUN` time on top of the already-local upstream base (see Speed) | +| running | ✓ | manual bridge smoke test (below) — a real agent-shaped sequence of `POST /execute` calls against task `692c77d_2` (id `1`) | +| isolation | ✓ | agent uid 1002/gid 0 denied `/appworld`, `ground_truth/*`, and `/tasks/1/id.txt`; `GET /evaluate` 404s (no such route exists on the bridge at all); `kill -USR1` on the bridge's pid as uid 1002 → `Operation not permitted` | +| reward-hacking | ✓ | grading reuses the bridge's own live `AppWorld` object rather than trusting anything the agent reports; the reward path is written by a root-owned process the agent has no route to influence (no `/evaluate` HTTP route, no signal permission) | +| oracle | ✓ | gold = 1.0 / no-op = 0.0, confirmed live this session (see Notes) | +| traces-reviewed | ✗ | the two existing fixtures (`tests/run/replay/fixtures/appworld-{292-terminus-2,584-claude-code}.traces.jsonl`, 23/49 spans) predate this fix — they were almost certainly recorded against the old runtime, which had no data, no bridge, and no way to act, so the agent gave up immediately. They should be re-recorded against the fixed image; see Notes | +| replicate-official | ? | not attempted — would mean running the upstream `appworld` reference agent and comparing its reported score to this container's `reward.txt` | + +## Safety — can the run harm us or cheat? + +| Check | Status | Evidence | +|-------|:------:|----------| +| egress-blocked | ? | not audited per-benchmark; AppWorld itself needs no runtime network (data is baked at build time) | +| agent-nonroot | ✓ | agent runs as uid 1002 (this benchmark's own `/entrypoint.sh`, not the shared launcher — see Notes) | +| secrets-isolated | n/a | no LLM credentials touch this container's filesystem; the gateway holds those | +| ground-truth-isolated | ✓ | `chmod -R go-rwx $APPWORLD_ROOT` at build time; verified live as uid 1002/gid 0 — `ls`/`cat` on `/appworld` and any `ground_truth/*.json` → Permission denied | +| task-identity-hidden | ✓ | `/tasks/$EVAL_TASK_ID/id.txt` stays root:600; verified live as uid 1002 → Permission denied. The real id is scoped to the bridge's own subprocess env (`APPWORLD_TASK_ID=...`), never exported broadly | +| grading-unreachable | ✓ | the bridge exposes exactly `GET /health` and `POST /execute`; `GET /evaluate` → 404. Grading is triggered by `SIGUSR1`, and signal delivery is uid-gated by the kernel — uid 1002 sending it to the root-owned bridge pid → `Operation not permitted` (verified live) | +| resource-limited | ? | not audited here — see `compose.yaml` for the shared `compose/services.yaml` limits | + +## Size + +| Metric | Value | +|--------|-------| +| image | 1.06 GB (`docker images`; `docker image inspect .Size` under-reports on this multi-platform/attestation build — do not trust that field) | +| per-task multiplier | shared-env (×1) — all 732 tasks' data is baked into one image at build time | + +## Speed + +| Metric | Value | +|--------|-------| +| build (no-cache `RUN` time, base image already local) | 28s total — `apt-get`: 7.1s, `pip install pyarrow`: 6.0s, HF task-metadata fetch: 2.0s, `appworld install` + `download data` (~193MB): 12.1s, permission/copy steps: <1s each | +| container start → bridge healthy | 9s (`/eval-materialize-task` + cold `AppWorld()` open, confirmed 4-5s cold per appworld's own timing) | +| grade (`SIGUSR1` → `evaluate()` → `reward.txt` written) | <1s — the bridge already holds the live session in memory, no reload needed | + +## Cost + +| Metric | Value | +|--------|-------| +| per task | no LLM cost from the benchmark image itself (all APIs are simulated, in-process); cost is entirely the agent's own token usage | +| full suite | 732 tasks × agent token cost; no benchmark-side compute cost beyond the ~10s/task container startup above | + +## Distribution — is it shipped? + +| Check | Status / Value | Evidence | +|-------|:--------------:|----------| +| published | ✓ | `ghcr.io/exgentic/benchmarks/appworld:latest` resolves (`docker manifest inspect`) | +| released label | ◐ | `LABEL eval.benchmark.released="true"` is present and fixtures do exist at the right path (rule 21a), but those fixtures predate this fix (see traces-reviewed above) — the label's *precondition* (a fixture proven against a working runtime) is currently stale | +| pull size | 1.06 GB | `docker images` | + +## Notes + +**The bug this audit fixes.** `output/appworld/1` showed `reward=-1, passed=false` for +every one of the 732 tasks — not because any task failed, but because there was no +AppWorld runtime for the agent to talk to at all: no task data installed, no server +running, and `/grade.sh` hardcoded `-1`. `-1` is this repo's convention for +*genuinely* externally-graded benchmarks (rule 18/20), but `world.evaluate()` is a +fully local, offline, in-process check — this was a mischaracterization, not a +real external-grading constraint. + +**Root cause of a second, more subtle bug found while fixing the first.** +AppWorld's own `AppWorld.__init__` → `initialize()` → `_prepare_directories()` +(`appworld/environment.py`) does `shutil.rmtree(self.output_directory, ...)` on +every construction — opening a *second* `AppWorld(task_id=..., experiment_name=...)` +silently discards whatever the first one did, even for the same task id and +experiment name, even within the same process. An earlier version of this fix +opened a fresh `AppWorld` session per `/execute` call (to force a flush-to-disk +after every call) and per grading pass (in a separate `grade.py` process) — this +looked reasonable but is actively wrong: every solve, correct or not, always +graded 0.0, which is a worse failure mode than the original bug because it looks +like it works. The official usage pattern (confirmed in `appworld/cli.py`) is one +long-lived session for the whole task — `execute()` already persists to disk +after every call internally, so the fix is to keep exactly one `AppWorld` object +alive for the bridge's lifetime and reuse it for grading too, rather than opening +a second one. + +**Grading channel: signal, not HTTP.** The agent shares the container's network +namespace with the bridge, so any TCP port bound to `127.0.0.1` is reachable to +it regardless of uid — there is no way to bind an HTTP route that only root can +reach. Signals don't have this problem: the kernel enforces that only a matching +uid (or a capable one) may `kill()` a process, so `/grade.sh` (root, run after the +agent's own process has exited) sending the root-owned bridge `SIGUSR1` is a route +the agent's uid 1002 genuinely cannot use — verified live (`Operation not +permitted`). + +**Oracle re-run (this image, task `692c77d_2` / id `1`).** No-op (bridge started, +nothing executed, `/grade.sh` run immediately): `reward.txt = 0.0`, `evaluate()` +returns its full 7-assertion breakdown to `/logs/appworld/evaluation.json` for +audit. Genuine solve (real `apis.spotify.*` calls over `/execute`: paginate the +14-song library and 11 liked songs, compute the 7 unliked, update the 2 that +already had a review to rating 1, add reviews for the other 5, then +`apis.supervisor.complete_task()`): `reward.txt = 1.0`, all 7 assertions pass. +Confirmed a *separate* `/execute` call (simulating the agent's next turn) sees +the previous call's mutations — the persistence bug above is fixed, not just +worked around. + +**Residuals (open):** +- **Stale replay fixtures.** The two fixtures backing the `released` label + (`tests/run/replay/fixtures/appworld-292-terminus-2.traces.jsonl`, + `appworld-584-claude-code.traces.jsonl`) are small (23 and 49 spans) and were + almost certainly recorded against the pre-fix image, where the agent had + nothing to do but give up. `tests/run/replay/test.rs`'s `assert_result_valid` + only checks structural shape (reward in range, required fields present), not + benchmark-specific semantics, so replaying these old fixtures against the new + image won't fail — but they also don't demonstrate anything about the fixed + runtime. These should be re-recorded against a real agent run before leaning + on them as evidence of end-to-end correctness. +- **Bypasses the shared framework launcher.** This benchmark's `/entrypoint.sh` + and `/grade.sh` are bespoke (predates this fix) rather than going through + `/usr/local/bin/run`/`run-agent` (rule 12/22). Left as-is: migrating to the + shared launcher is an unrelated, larger refactor and wasn't part of this fix's + scope. +- **`egress-blocked` / `resource-limited` / `replicate-official`** — not audited + in this pass; see `compose.yaml` for the shared network/resource config this + benchmark inherits from `compose/services.yaml`. diff --git a/containers/benchmarks/appworld/Dockerfile b/containers/benchmarks/appworld/Dockerfile index 7dc3c8bf..48f7bad4 100644 --- a/containers/benchmarks/appworld/Dockerfile +++ b/containers/benchmarks/appworld/Dockerfile @@ -1,8 +1,18 @@ # AppWorld # 732 tasks across 9 simulated apps. Data: HuggingFace LukaszTP/AppWorld-Tasks # Upstream image: ghcr.io/stonybrooknlp/appworld — includes all APIs in-process. -# Evaluation is state-based (database checks), requires the AppWorld runtime. -# Externally graded: test writes -1. +# Evaluation is state-based (database checks) and fully local/offline +# (world.evaluate()) — graded for real, not externally. +# +# The agent never gets in-process SDK access (that would expose +# ground_truth/answer.json, which sits next to the dbs/*.jsonl files it +# legitimately needs) and never learns its real task id (rule 7). Instead, +# entrypoint.sh starts bridge.py in the background, a root-owned HTTP +# service bound to 127.0.0.1 that holds the one live AppWorld session and +# exposes only a sanitized POST /execute {"code": ...} route. /grade.sh runs +# later, as root, after the agent's process has exited: it signals bridge.py +# (SIGUSR1, not an HTTP route — signals respect uid, so this stays +# unreachable to the agent) to evaluate that same session directly. ARG REGISTRY=ghcr.io/exgentic ARG REGISTRY_SUFFIX=/ @@ -48,6 +58,23 @@ PYEOF RUN chmod 600 /tasks/all.jsonl +# Install the bundled apps/tests and download the full task dataset + base +# app DBs at build time (no network needed at container runtime). ~193M, +# ~15s, no auth required — verified live against this same upstream image. +ENV APPWORLD_ROOT=/appworld +RUN /opt/venv/bin/python -m appworld.cli install --no-repo \ + && /opt/venv/bin/python -m appworld.cli download data --root "$APPWORLD_ROOT" + +# Root-only: each task directory's ground_truth/{answer,private_data, +# test_data}.json and evaluation.py sit right next to the dbs/*.jsonl the +# agent legitimately needs — there is no way to grant dbs/ access without +# also granting ground_truth/ access at the filesystem level. The agent +# (uid 1002, gid 0 — same group as root, see combination.Dockerfile) must +# never read this tree directly; `go-rwx` strips group bits too, since a +# shared gid alone isn't enough (see /tasks/*/id.txt using the same +# root:600 pattern for the same reason). +RUN chmod -R go-rwx "$APPWORLD_ROOT" + WORKDIR /app # openclaw (and any agent that writes into its cwd, e.g. AGENTS.md) runs as @@ -55,12 +82,27 @@ WORKDIR /app # /app, which otherwise stays root:root 755 from the upstream image. RUN chown -R 1002:0 /app && chmod -R g+rwX /app -# Test: AppWorld evaluation is state-based (database checks) and requires -# the full AppWorld runtime. This is externally graded — always write -1. +COPY bridge.py / +RUN chmod 700 /bridge.py + +# Test: AppWorld evaluation is state-based (database checks) but fully +# local/offline (world.evaluate()). Grading reuses the same live AppWorld +# object bridge.py has been mutating rather than opening a second one — +# AppWorld() re-initializes by deleting its own output directory on +# construction, so a second open here would just discard the agent's +# progress. SIGUSR1 (not an HTTP route) triggers it: only root can signal a +# root-owned process, so this stays unreachable to the agent's uid even +# though it shares the container's network namespace with the bridge. RUN mkdir -p /tests && cat > /grade.sh <<'TEST' #!/bin/bash -mkdir -p /logs/verifier -echo -1 > /logs/verifier/reward.txt +mkdir -p /logs/verifier /logs/appworld +kill -USR1 "$(cat /run/bridge.pid)" +for i in $(seq 1 150); do + if [ -f /logs/verifier/reward.txt ]; then + break + fi + sleep 0.2 +done TEST RUN chmod +x /grade.sh @@ -71,8 +113,43 @@ RUN chmod +x /eval-materialize-task RUN cat > /entrypoint.sh <<'ENTRY' #!/bin/bash /eval-materialize-task + +REAL_TASK_ID="$(cat /tasks/$EVAL_TASK_ID/id.txt)" +APPWORLD_TASK_ID="$REAL_TASK_ID" /opt/venv/bin/python /bridge.py & +echo $! > /run/bridge.pid + +for i in $(seq 1 150); do + if curl -sf http://127.0.0.1:8123/health >/dev/null 2>&1; then + break + fi + sleep 0.2 +done + if [ -n "$EVAL_TASK_ID" ] && [ -z "$TASK" ]; then - export TASK="$(cat /tasks/$EVAL_TASK_ID/problem.txt)" + BOILERPLATE="You are solving a task in the AppWorld simulated environment +(9 apps, 457 APIs: spotify, amazon, gmail, phone, file_system, simple_note, +todoist, venmo, splitwise, plus supervisor/api_docs/admin). + +A code-execution bridge is running locally. To run Python code against the +simulated apps (an \`apis\` object is bound for you), send a POST request to +http://127.0.0.1:8123/execute with a JSON body: {\"code\": \"\"}. + +Example: +python3 -c 'import json, urllib.request +req = urllib.request.Request(\"http://127.0.0.1:8123/execute\", + data=json.dumps({\"code\": \"print(apis.api_docs.show_app_descriptions())\"}).encode(), + headers={\"Content-Type\": \"application/json\"}) +print(urllib.request.urlopen(req).read().decode())' + +The response is JSON: {\"output\": \"\"} on +success, or {\"error\": \"\"} on failure. Variables you define persist +across calls within this task. Call apis.supervisor.complete_task() via this +bridge once you believe the task is complete — after that, no further +actions are graded. + +Your task: +" + export TASK="${BOILERPLATE}$(cat /tasks/$EVAL_TASK_ID/problem.txt)" fi exec "$@" ENTRY diff --git a/containers/benchmarks/appworld/README.md b/containers/benchmarks/appworld/README.md index 677a6f7e..db0c511a 100644 --- a/containers/benchmarks/appworld/README.md +++ b/containers/benchmarks/appworld/README.md @@ -1,6 +1,6 @@ # appworld -**Status:** Released ✓ — sample trajectory: [`tests/fixtures/appworld-0-claude-code.traces.jsonl`](../../tests/fixtures/appworld-0-claude-code.traces.jsonl) +**Status:** Released ✓ — sample trajectory: [`tests/run/replay/fixtures/appworld-292-terminus-2.traces.jsonl`](../../tests/run/replay/fixtures/appworld-292-terminus-2.traces.jsonl) AppWorld - 9 simulated apps with 457 APIs @@ -19,11 +19,35 @@ AppWorld - 9 simulated apps with 457 APIs ## What the agent sees -The agent receives a task of the form: "$(cat /tasks/$EVAL_TASK_ID/problem.txt)"" The problem text is read from `/tasks/$EVAL_TASK_ID/problem.txt` and passed in via the `TASK` environment variable. +The agent receives a task in `TASK` made up of two parts: + +1. Generic boilerplate explaining that a code-execution bridge is running + locally at `http://127.0.0.1:8123/execute` — `POST {"code": "..."}` runs + Python against the simulated apps (the `apis` object is bound), returning + `{"output": ...}` / `{"error": ...}`. Variables persist across calls; the + agent calls `apis.supervisor.complete_task()` when done. +2. The task instruction itself, read from `/tasks/$EVAL_TASK_ID/problem.txt`. + +The agent never gets direct SDK/filesystem access to AppWorld and never +learns its real task id (`EVAL_TASK_ID`/`TASK_ID` are excluded from its +environment, per rule 7) — `bridge.py`, a root-owned background HTTP service +started by `/entrypoint.sh`, holds the one live AppWorld session for the +real task id and proxies only sanitized code execution. This also keeps the +agent away from each task's `ground_truth/{answer,private_data,test_data}.json` +and `evaluation.py`, which sit right next to the `dbs/*.jsonl` files it +legitimately needs to interact with. ## How it's graded -Custom `/grade.sh` defined inline in the Dockerfile. Reward is hard-coded to `-1` inside the container — this benchmark is externally graded (e.g. LLM-as-judge or uploaded to a leaderboard). +`world.evaluate()` — AppWorld's own state-based, fully local/offline check +(database assertions against the task's `ground_truth/`). `AppWorld()` +re-initializes by deleting its own output directory on construction, so +grading reopening a second session would discard the agent's progress; +instead `/grade.sh` runs as root after the agent's process has exited and +sends `bridge.py` `SIGUSR1` (not an HTTP route — signals respect uid, so +this stays unreachable to the agent) to evaluate the same live session it's +been mutating. Reward is `1.0` if `evaluate().success`, else `0.0` — not +externally graded. ## Files diff --git a/containers/benchmarks/appworld/bridge.py b/containers/benchmarks/appworld/bridge.py new file mode 100644 index 00000000..d555c9d7 --- /dev/null +++ b/containers/benchmarks/appworld/bridge.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +# Root-owned code-execution bridge for the AppWorld runtime. Runs in the +# background from /entrypoint.sh, before the agent (uid 1002) starts. +# +# Exposes exactly two HTTP routes on 127.0.0.1: GET /health and POST +# /execute. Deliberately does NOT expose evaluate() over HTTP — failing +# AppWorld assertions embed the literal expected values in their message, so +# a reachable /evaluate route would let the agent read the answer, and the +# agent shares this container's network namespace so any bound TCP port is +# reachable to it regardless of uid. +# +# AppWorld() re-initializes by deleting its own output directory (see +# _prepare_directories in appworld/environment.py) — opening a fresh session +# per call, as an earlier version of this file did, silently discards every +# prior call's state. The correct usage (matching appworld's own cli.py) is +# one long-lived session for the whole task: execute() already persists to +# disk after every call internally, so this process just needs to keep that +# one `world` object alive and call execute() on it repeatedly. +# +# Grading reuses that same live object instead of opening a second one, and +# is triggered by SIGUSR1 rather than an HTTP route: signals respect uid +# (only root can signal a root-owned process), so /grade.sh (root, run after +# the agent's own process has exited) can ask for evaluation without the +# agent ever having a network path to it. + +import json +import os +import signal +import sys +import traceback +from http.server import BaseHTTPRequestHandler, HTTPServer + +from appworld import AppWorld + +TASK_ID = os.environ["APPWORLD_TASK_ID"] +EXPERIMENT_NAME = "agent" +PORT = int(os.environ.get("APPWORLD_BRIDGE_PORT", "8123")) + +world = None + + +def handle_evaluate_signal(signum, frame): + try: + result = world.evaluate().to_dict() + except Exception as e: + result = {"success": False, "error": f"{type(e).__name__}: {e}"} + + os.makedirs("/logs/appworld", exist_ok=True) + os.makedirs("/logs/verifier", exist_ok=True) + with open("/logs/appworld/evaluation.json", "w") as f: + json.dump(result, f, indent=2, default=str) + reward = 1.0 if result.get("success") else 0.0 + with open("/logs/verifier/reward.txt", "w") as f: + f.write(str(reward)) + sys.exit(0) + + +class Handler(BaseHTTPRequestHandler): + def _send_json(self, status, payload): + body = json.dumps(payload).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): + if self.path == "/health": + self._send_json(200, {"status": "ok"}) + else: + self._send_json(404, {"error": "not found"}) + + def do_POST(self): + if self.path != "/execute": + self._send_json(404, {"error": "not found"}) + return + length = int(self.headers.get("Content-Length", 0)) + try: + body = json.loads(self.rfile.read(length) or b"{}") + code = body["code"] + except Exception: + self._send_json(400, {"error": 'expected JSON body: {"code": "..."}'}) + return + try: + output = world.execute(code) + self._send_json(200, {"output": output}) + except Exception as e: + self._send_json(200, {"error": f"{type(e).__name__}: {e}"}) + + def log_message(self, format, *args): + pass + + +def main(): + global world + signal.signal(signal.SIGUSR1, handle_evaluate_signal) + + # Opened once, kept alive for the process lifetime (see module docstring + # for why re-opening per call is wrong). First open is 4-5s, so /health + # only reports ready once this has actually completed. + try: + world = AppWorld(task_id=TASK_ID, experiment_name=EXPERIMENT_NAME) + except Exception: + traceback.print_exc(file=sys.stderr) + sys.exit(1) + + # Single-threaded: AppWorld's sqlite connections are bound to the thread + # that created them, and a threading server would call world.execute() + # from a different thread than the one that opened `world` above. One + # agent, sequential turns, doesn't need concurrency. + server = HTTPServer(("127.0.0.1", PORT), Handler) + server.serve_forever() + + +if __name__ == "__main__": + main()