fix(appworld): grade via persistent AppWorld session, not hardcoded -1 - #352
Open
cdoron wants to merge 1 commit into
Open
fix(appworld): grade via persistent AppWorld session, not hardcoded -1#352cdoron wants to merge 1 commit into
cdoron wants to merge 1 commit into
Conversation
cdoron
force-pushed
the
fix/appworld-bridge-persistent-grading
branch
2 times, most recently
from
August 12, 2026 08:21
cb065fe to
490584f
Compare
AppWorld's evaluation is fully local/offline (world.evaluate()), but
/grade.sh previously hardcoded reward=-1 as if it were externally graded,
so every one of the 732 tasks scored -1 regardless of the agent's actions.
Fix: bridge.py holds one long-lived AppWorld session for the task, started
by /entrypoint.sh before the agent runs, exposing only a sanitized
POST /execute {"code": ...} route over HTTP. AppWorld() destructively
re-initializes on construction, so grading reuses that same live session
via SIGUSR1 (root-only, since the agent shares the container's network
namespace and any HTTP route would be reachable to it regardless of uid)
rather than opening a second session that would discard the agent's
progress.
Verified live: oracle re-run on task 692c77d_2 scores 0.0 on a no-op and
1.0 on a genuine solve, with isolation/reward-hacking checks confirming
the agent's uid can't reach /evaluate or signal the bridge itself. See
AUDIT.md for full evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Doron Chen <cdoron@il.ibm.com>
cdoron
force-pushed
the
fix/appworld-bridge-persistent-grading
branch
from
August 12, 2026 08:25
490584f to
bba0d7d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
world.evaluate()), but/grade.shhardcodedreward=-1as if it were externally graded, so every one of the 732 tasks scored-1regardless of the agent's actions — and the agent had no way to call AppWorld's APIs at all (no task data installed, no server running).bridge.py: a root-owned HTTP service holding one long-livedAppWorldsession per task, exposing a sanitizedPOST /execute {"code": ...}route.AppWorld()destructively re-initializes on construction, so grading reuses that same live session viaSIGUSR1(root-only — the agent shares the container's network namespace, so any HTTP route would be reachable to it regardless of uid) rather than opening a second session that would discard the agent's progress.README.mdto describe the new task boilerplate/bridge mechanism, and addsAUDIT.mddocumenting live verification (oracle re-run scoring 0.0 on a no-op / 1.0 on a genuine solve, isolation and reward-hacking checks).Test plan
docker build --platform linux/amd64 --no-cachesucceeds692c77d_2: no-op →reward=0.0; genuine solve (realapis.spotify.*calls) →reward=1.0/appworld,ground_truth/*,/tasks/<id>/id.txtall permission-denied;GET /evaluate404s;kill -USR1on the bridge as uid 1002 →Operation not permittedtests/run/replay/fixtures/appworld-{292-terminus-2,584-claude-code}.traces.jsonlagainst the fixed image (predate this fix, flagged as a residual in AUDIT.md — not blocking this PR)🤖 Generated with Claude Code