Exgentic · cdoron · Aug 12, 2026
diff --git a/containers/benchmarks/appworld/AUDIT.md b/containers/benchmarks/appworld/AUDIT.md
@@ -0,0 +1,128 @@
+---
+benchmark: appworld
+host: local Colima (amd64 via Rosetta, no QEMU)
+commit: d9ef64a9
+---
+# Audit — appworld
+
+`✓` verified (a check passed) · `◐` partial (holds in one surface, open in another) · `✗` failing · `?` unchecked · `n/a` not applicable
+
+## Validity — is the score real?
+
+| Check | Status | Evidence |
+|-------|:------:|----------|
+| building | ✓ | `docker build --platform linux/amd64` succeeds, no-cache, 28s of `RUN` time on top of the already-local upstream base (see Speed) |
+| running | ✓ | manual bridge smoke test (below) — a real agent-shaped sequence of `POST /execute` calls against task `692c77d_2` (id `1`) |
+| isolation | ✓ | agent uid 1002/gid 0 denied `/appworld`, `ground_truth/*`, and `/tasks/1/id.txt`; `GET /evaluate` 404s (no such route exists on the bridge at all); `kill -USR1` on the bridge's pid as uid 1002 → `Operation not permitted` |
+| reward-hacking | ✓ | grading reuses the bridge's own live `AppWorld` object rather than trusting anything the agent reports; the reward path is written by a root-owned process the agent has no route to influence (no `/evaluate` HTTP route, no signal permission) |
+| oracle | ✓ | gold = 1.0 / no-op = 0.0, confirmed live this session (see Notes) |
+| traces-reviewed | ✗ | the two existing fixtures (`tests/run/replay/fixtures/appworld-{292-terminus-2,584-claude-code}.traces.jsonl`, 23/49 spans) predate this fix — they were almost certainly recorded against the old runtime, which had no data, no bridge, and no way to act, so the agent gave up immediately. They should be re-recorded against the fixed image; see Notes |
+| replicate-official | ? | not attempted — would mean running the upstream `appworld` reference agent and comparing its reported score to this container's `reward.txt` |
+
+## Safety — can the run harm us or cheat?
+
+| Check | Status | Evidence |
+|-------|:------:|----------|
+| egress-blocked | ? | not audited per-benchmark; AppWorld itself needs no runtime network (data is baked at build time) |
+| agent-nonroot | ✓ | agent runs as uid 1002 (this benchmark's own `/entrypoint.sh`, not the shared launcher — see Notes) |
+| secrets-isolated | n/a | no LLM credentials touch this container's filesystem; the gateway holds those |
+| ground-truth-isolated | ✓ | `chmod -R go-rwx $APPWORLD_ROOT` at build time; verified live as uid 1002/gid 0 — `ls`/`cat` on `/appworld` and any `ground_truth/*.json` → Permission denied |
+| task-identity-hidden | ✓ | `/tasks/$EVAL_TASK_ID/id.txt` stays root:600; verified live as uid 1002 → Permission denied. The real id is scoped to the bridge's own subprocess env (`APPWORLD_TASK_ID=...`), never exported broadly |
+| grading-unreachable | ✓ | the bridge exposes exactly `GET /health` and `POST /execute`; `GET /evaluate` → 404. Grading is triggered by `SIGUSR1`, and signal delivery is uid-gated by the kernel — uid 1002 sending it to the root-owned bridge pid → `Operation not permitted` (verified live) |
+| resource-limited | ? | not audited here — see `compose.yaml` for the shared `compose/services.yaml` limits |
+
+## Size
+
+| Metric | Value |
+|--------|-------|
+| image | 1.06 GB (`docker images`; `docker image inspect .Size` under-reports on this multi-platform/attestation build — do not trust that field) |
+| per-task multiplier | shared-env (×1) — all 732 tasks' data is baked into one image at build time |
+
+## Speed
+
+| Metric | Value |
+|--------|-------|
+| build (no-cache `RUN` time, base image already local) | 28s total — `apt-get`: 7.1s, `pip install pyarrow`: 6.0s, HF task-metadata fetch: 2.0s, `appworld install` + `download data` (~193MB): 12.1s, permission/copy steps: <1s each |
+| container start → bridge healthy | 9s (`/eval-materialize-task` + cold `AppWorld()` open, confirmed 4-5s cold per appworld's own timing) |
+| grade (`SIGUSR1` → `evaluate()` → `reward.txt` written) | <1s — the bridge already holds the live session in memory, no reload needed |
+
+## Cost
+
+| Metric | Value |
+|--------|-------|
+| per task | no LLM cost from the benchmark image itself (all APIs are simulated, in-process); cost is entirely the agent's own token usage |
+| full suite | 732 tasks × agent token cost; no benchmark-side compute cost beyond the ~10s/task container startup above |
+
+## Distribution — is it shipped?
+
+| Check | Status / Value | Evidence |
+|-------|:--------------:|----------|
+| published | ✓ | `ghcr.io/exgentic/benchmarks/appworld:latest` resolves (`docker manifest inspect`) |
+| released label | ◐ | `LABEL eval.benchmark.released="true"` is present and fixtures do exist at the right path (rule 21a), but those fixtures predate this fix (see traces-reviewed above) — the label's *precondition* (a fixture proven against a working runtime) is currently stale |
+| pull size | 1.06 GB | `docker images` |
+
+## Notes
+
+**The bug this audit fixes.** `output/appworld/1` showed `reward=-1, passed=false` for
+every one of the 732 tasks — not because any task failed, but because there was no
+AppWorld runtime for the agent to talk to at all: no task data installed, no server
+running, and `/grade.sh` hardcoded `-1`. `-1` is this repo's convention for
+*genuinely* externally-graded benchmarks (rule 18/20), but `world.evaluate()` is a
+fully local, offline, in-process check — this was a mischaracterization, not a
+real external-grading constraint.
+
+**Root cause of a second, more subtle bug found while fixing the first.**
+AppWorld's own `AppWorld.__init__` → `initialize()` → `_prepare_directories()`
+(`appworld/environment.py`) does `shutil.rmtree(self.output_directory, ...)` on
+every construction — opening a *second* `AppWorld(task_id=..., experiment_name=...)`
+silently discards whatever the first one did, even for the same task id and
+experiment name, even within the same process. An earlier version of this fix
+opened a fresh `AppWorld` session per `/execute` call (to force a flush-to-disk
+after every call) and per grading pass (in a separate `grade.py` process) — this
+looked reasonable but is actively wrong: every solve, correct or not, always
+graded 0.0, which is a worse failure mode than the original bug because it looks
+like it works. The official usage pattern (confirmed in `appworld/cli.py`) is one
+long-lived session for the whole task — `execute()` already persists to disk
+after every call internally, so the fix is to keep exactly one `AppWorld` object
+alive for the bridge's lifetime and reuse it for grading too, rather than opening
+a second one.
+
+**Grading channel: signal, not HTTP.** The agent shares the container's network
+namespace with the bridge, so any TCP port bound to `127.0.0.1` is reachable to
+it regardless of uid — there is no way to bind an HTTP route that only root can
+reach. Signals don't have this problem: the kernel enforces that only a matching
+uid (or a capable one) may `kill()` a process, so `/grade.sh` (root, run after the
+agent's own process has exited) sending the root-owned bridge `SIGUSR1` is a route
+the agent's uid 1002 genuinely cannot use — verified live (`Operation not
+permitted`).
+
+**Oracle re-run (this image, task `692c77d_2` / id `1`).** No-op (bridge started,
+nothing executed, `/grade.sh` run immediately): `reward.txt = 0.0`, `evaluate()`
+returns its full 7-assertion breakdown to `/logs/appworld/evaluation.json` for
+audit. Genuine solve (real `apis.spotify.*` calls over `/execute`: paginate the
+14-song library and 11 liked songs, compute the 7 unliked, update the 2 that
+already had a review to rating 1, add reviews for the other 5, then
+`apis.supervisor.complete_task()`): `reward.txt = 1.0`, all 7 assertions pass.
+Confirmed a *separate* `/execute` call (simulating the agent's next turn) sees
+the previous call's mutations — the persistence bug above is fixed, not just
+worked around.
+
+**Residuals (open):**
+- **Stale replay fixtures.** The two fixtures backing the `released` label
+  (`tests/run/replay/fixtures/appworld-292-terminus-2.traces.jsonl`,
+  `appworld-584-claude-code.traces.jsonl`) are small (23 and 49 spans) and were
+  almost certainly recorded against the pre-fix image, where the agent had
+  nothing to do but give up. `tests/run/replay/test.rs`'s `assert_result_valid`
+  only checks structural shape (reward in range, required fields present), not
+  benchmark-specific semantics, so replaying these old fixtures against the new
+  image won't fail — but they also don't demonstrate anything about the fixed
+  runtime. These should be re-recorded against a real agent run before leaning
+  on them as evidence of end-to-end correctness.
+- **Bypasses the shared framework launcher.** This benchmark's `/entrypoint.sh`
+  and `/grade.sh` are bespoke (predates this fix) rather than going through
+  `/usr/local/bin/run`/`run-agent` (rule 12/22). Left as-is: migrating to the
+  shared launcher is an unrelated, larger refactor and wasn't part of this fix's
+  scope.
+- **`egress-blocked` / `resource-limited` / `replicate-official`** — not audited
+  in this pass; see `compose.yaml` for the shared network/resource config this
+  benchmark inherits from `compose/services.yaml`.
diff --git a/containers/benchmarks/appworld/Dockerfile b/containers/benchmarks/appworld/Dockerfile
@@ -1,8 +1,18 @@
 # AppWorld
 # 732 tasks across 9 simulated apps. Data: HuggingFace LukaszTP/AppWorld-Tasks
 # Upstream image: ghcr.io/stonybrooknlp/appworld — includes all APIs in-process.
-# Evaluation is state-based (database checks), requires the AppWorld runtime.
-# Externally graded: test writes -1.
+# Evaluation is state-based (database checks) and fully local/offline
+# (world.evaluate()) — graded for real, not externally.
+#
+# The agent never gets in-process SDK access (that would expose
+# ground_truth/answer.json, which sits next to the dbs/*.jsonl files it
+# legitimately needs) and never learns its real task id (rule 7). Instead,
+# entrypoint.sh starts bridge.py in the background, a root-owned HTTP
+# service bound to 127.0.0.1 that holds the one live AppWorld session and
+# exposes only a sanitized POST /execute {"code": ...} route. /grade.sh runs
+# later, as root, after the agent's process has exited: it signals bridge.py
+# (SIGUSR1, not an HTTP route — signals respect uid, so this stays
+# unreachable to the agent) to evaluate that same session directly.
 
 ARG REGISTRY=ghcr.io/exgentic
 ARG REGISTRY_SUFFIX=/
@@ -48,19 +58,51 @@ PYEOF
 
 RUN chmod 600 /tasks/all.jsonl
 
+# Install the bundled apps/tests and download the full task dataset + base
+# app DBs at build time (no network needed at container runtime). ~193M,
+# ~15s, no auth required — verified live against this same upstream image.
+ENV APPWORLD_ROOT=/appworld
+RUN /opt/venv/bin/python -m appworld.cli install --no-repo \
+    && /opt/venv/bin/python -m appworld.cli download data --root "$APPWORLD_ROOT"
+
+# Root-only: each task directory's ground_truth/{answer,private_data,
+# test_data}.json and evaluation.py sit right next to the dbs/*.jsonl the
+# agent legitimately needs — there is no way to grant dbs/ access without
+# also granting ground_truth/ access at the filesystem level. The agent
+# (uid 1002, gid 0 — same group as root, see combination.Dockerfile) must
+# never read this tree directly; `go-rwx` strips group bits too, since a
+# shared gid alone isn't enough (see /tasks/*/id.txt using the same
+# root:600 pattern for the same reason).
+RUN chmod -R go-rwx "$APPWORLD_ROOT"
+
 WORKDIR /app
 
 # openclaw (and any agent that writes into its cwd, e.g. AGENTS.md) runs as
 # uid 1002 (containers/core/combination.Dockerfile); grant it write access to
 # /app, which otherwise stays root:root 755 from the upstream image.
 RUN chown -R 1002:0 /app && chmod -R g+rwX /app
 
-# Test: AppWorld evaluation is state-based (database checks) and requires
-# the full AppWorld runtime. This is externally graded — always write -1.
+COPY bridge.py /
+RUN chmod 700 /bridge.py
+
+# Test: AppWorld evaluation is state-based (database checks) but fully
+# local/offline (world.evaluate()). Grading reuses the same live AppWorld
+# object bridge.py has been mutating rather than opening a second one —
+# AppWorld() re-initializes by deleting its own output directory on
+# construction, so a second open here would just discard the agent's
+# progress. SIGUSR1 (not an HTTP route) triggers it: only root can signal a
+# root-owned process, so this stays unreachable to the agent's uid even
+# though it shares the container's network namespace with the bridge.
 RUN mkdir -p /tests && cat > /grade.sh <<'TEST'
 #!/bin/bash
-mkdir -p /logs/verifier
-echo -1 > /logs/verifier/reward.txt
+mkdir -p /logs/verifier /logs/appworld
+kill -USR1 "$(cat /run/bridge.pid)"
+for i in $(seq 1 150); do
+  if [ -f /logs/verifier/reward.txt ]; then
+    break
+  fi
+  sleep 0.2
+done
 TEST
 
 RUN chmod +x /grade.sh
@@ -71,8 +113,43 @@ RUN chmod +x /eval-materialize-task
 RUN cat > /entrypoint.sh <<'ENTRY'
 #!/bin/bash
 /eval-materialize-task
+
+REAL_TASK_ID="$(cat /tasks/$EVAL_TASK_ID/id.txt)"
+APPWORLD_TASK_ID="$REAL_TASK_ID" /opt/venv/bin/python /bridge.py &
+echo $! > /run/bridge.pid
+
+for i in $(seq 1 150); do
+  if curl -sf http://127.0.0.1:8123/health >/dev/null 2>&1; then
+    break
+  fi
+  sleep 0.2
+done
+
 if [ -n "$EVAL_TASK_ID" ] && [ -z "$TASK" ]; then
-  export TASK="$(cat /tasks/$EVAL_TASK_ID/problem.txt)"
+  BOILERPLATE="You are solving a task in the AppWorld simulated environment
+(9 apps, 457 APIs: spotify, amazon, gmail, phone, file_system, simple_note,
+todoist, venmo, splitwise, plus supervisor/api_docs/admin).
+
+A code-execution bridge is running locally. To run Python code against the
+simulated apps (an \`apis\` object is bound for you), send a POST request to
+http://127.0.0.1:8123/execute with a JSON body: {\"code\": \"<python source>\"}.
+
+Example:
+python3 -c 'import json, urllib.request
+req = urllib.request.Request(\"http://127.0.0.1:8123/execute\",
+    data=json.dumps({\"code\": \"print(apis.api_docs.show_app_descriptions())\"}).encode(),
+    headers={\"Content-Type\": \"application/json\"})
+print(urllib.request.urlopen(req).read().decode())'
+
+The response is JSON: {\"output\": \"<stdout captured during execution>\"} on
+success, or {\"error\": \"<message>\"} on failure. Variables you define persist
+across calls within this task. Call apis.supervisor.complete_task() via this
+bridge once you believe the task is complete — after that, no further
+actions are graded.
+
+Your task:
+"
+  export TASK="${BOILERPLATE}$(cat /tasks/$EVAL_TASK_ID/problem.txt)"
 fi
 exec "$@"
 ENTRY

diff --git a/containers/benchmarks/appworld/README.md b/containers/benchmarks/appworld/README.md
@@ -1,6 +1,6 @@
 # appworld
 
-**Status:** Released ✓ — sample trajectory: [`tests/fixtures/appworld-0-claude-code.traces.jsonl`](../../tests/fixtures/appworld-0-claude-code.traces.jsonl)
+**Status:** Released ✓ — sample trajectory: [`tests/run/replay/fixtures/appworld-292-terminus-2.traces.jsonl`](../../tests/run/replay/fixtures/appworld-292-terminus-2.traces.jsonl)
 
 
 AppWorld - 9 simulated apps with 457 APIs
@@ -19,11 +19,35 @@ AppWorld - 9 simulated apps with 457 APIs
 
 ## What the agent sees
 
-The agent receives a task of the form: "$(cat /tasks/$EVAL_TASK_ID/problem.txt)"" The problem text is read from `/tasks/$EVAL_TASK_ID/problem.txt` and passed in via the `TASK` environment variable.
+The agent receives a task in `TASK` made up of two parts:
+
+1. Generic boilerplate explaining that a code-execution bridge is running
+   locally at `http://127.0.0.1:8123/execute` — `POST {"code": "..."}` runs
+   Python against the simulated apps (the `apis` object is bound), returning
+   `{"output": ...}` / `{"error": ...}`. Variables persist across calls; the
+   agent calls `apis.supervisor.complete_task()` when done.
+2. The task instruction itself, read from `/tasks/$EVAL_TASK_ID/problem.txt`.
+
+The agent never gets direct SDK/filesystem access to AppWorld and never
+learns its real task id (`EVAL_TASK_ID`/`TASK_ID` are excluded from its
+environment, per rule 7) — `bridge.py`, a root-owned background HTTP service
+started by `/entrypoint.sh`, holds the one live AppWorld session for the
+real task id and proxies only sanitized code execution. This also keeps the
+agent away from each task's `ground_truth/{answer,private_data,test_data}.json`
+and `evaluation.py`, which sit right next to the `dbs/*.jsonl` files it
+legitimately needs to interact with.
 
 ## How it's graded
 
-Custom `/grade.sh` defined inline in the Dockerfile. Reward is hard-coded to `-1` inside the container — this benchmark is externally graded (e.g. LLM-as-judge or uploaded to a leaderboard).
+`world.evaluate()` — AppWorld's own state-based, fully local/offline check
+(database assertions against the task's `ground_truth/`). `AppWorld()`
+re-initializes by deleting its own output directory on construction, so
+grading reopening a second session would discard the agent's progress;
+instead `/grade.sh` runs as root after the agent's process has exited and
+sends `bridge.py` `SIGUSR1` (not an HTTP route — signals respect uid, so
+this stays unreachable to the agent) to evaluate the same live session it's
+been mutating. Reward is `1.0` if `evaluate().success`, else `0.0` — not
+externally graded.
 
 ## Files