fix(swe-agent): point sweagent at /testbed instead of a fake /workspace/repo - #323
fix(swe-agent): point sweagent at /testbed instead of a fake /workspace/repo#323cdoron wants to merge 1 commit into
Conversation
…ce/repo The benchmark's actual checkout — at the buggy commit grade.sh diffs against — lives at /testbed, not /workspace. The old run.sh instead git-init'd an empty placeholder repo at /workspace/repo with a single empty commit, so SWE-agent could never have graded correctly: any patch it produced was a diff against nothing, not against the real bug. Switch --env.repo.repo_name to `testbed` (LocalRepoConfig's `preexisting` type is a no-op copy, so it can point directly at the existing checkout) and drop the /workspace/repo git-init scaffolding entirely. This also removes the need for the /workspace directory itself, so the build-time `mkdir -p /workspace && chown 1002:0 /workspace && chmod g+rwX /workspace` step (added to fix a permission error when creating it) is now dead code and is removed along with it. Verified with: eval-containers run swe-bench --agent swe-agent --model gcp/gemini-3.5-flash-lite --local --task-id astropy__astropy-12907 — agent edits land in /testbed and grade.sh's diff against the real bug now reflects the agent's actual patch (reward:1, passed:true). Checked against: .agents/benchmarks/RULES.md Signed-off-by: Doron Chen <cdoron@il.ibm.com>
elronbandel
left a comment
There was a problem hiding this comment.
Verified against grade.sh directly: it does cd /testbed; git diff — that's the real grading target, and it's what TASK already told the agent ("the repository at /testbed"). Before this fix sweagent operated on /workspace/repo, an empty git-init'd placeholder totally disconnected from /testbed — meaning every swe-agent/swe-bench run scored 0 unconditionally, and the TASK text contradicted the actual environment. This fix makes repo_name, safe.directory, and TASK all agree on /testbed. Confirmed no stale /workspace references remain in the file. This also fully obsoletes #318.
elronbandel
left a comment
There was a problem hiding this comment.
Walking back my earlier approval — the underlying bug and grading analysis were correct, but the fix is scoped too narrowly and breaks other benchmarks.
`containers/agents/swe-agent/Dockerfile` is the generic agent image, shared across every benchmark paired with swe-agent (agents/RULES.md rule 4: "the agent MUST NOT know which benchmark it is running in"). This PR hardcodes `--env.repo.repo_name=testbed` unconditionally. Checked the other benchmarks with existing swe-agent replay fixtures — `apps`, `code-contests`, `math-500`, `mmmu` — none of them have a `/testbed` directory (code-contests/math-500 use `/app`, apps/mmmu declare no WORKDIR at all). sweagent's `preexisting` repo type assumes the path already exists (no copy, no init) — pointed at a nonexistent `/testbed` for those four, this will very likely fail at startup, breaking `apps-2999-swe-agent`, `code-contests-98-swe-agent`, `math-500-299-swe-agent`, and `mmmu-359-swe-agent` replay tests.
The swe-bench-specific fix (pointing at /testbed) is correct for swe-bench, but needs to stay benchmark-agnostic — e.g. detect the actual cwd/WORKDIR at runtime the way #308 did for openclaw (`$(pwd)` instead of a hardcoded path), rather than hardcoding `testbed` into the shared agent image.
Separately (not this PR's fault): this now has a real merge conflict with main since #319 already landed independently and touches the same block.
The benchmark's actual checkout — at the buggy commit grade.sh diffs
against — lives at /testbed, not /workspace. The old run.sh instead
git-init'd an empty placeholder repo at /workspace/repo with a single
empty commit, so SWE-agent could never have graded correctly: any patch
it produced was a diff against nothing, not against the real bug.
Switch --env.repo.repo_name to
testbed(LocalRepoConfig'spreexistingtype is a no-op copy, so it can point directly at the existing checkout)
and drop the /workspace/repo git-init scaffolding entirely. This also
removes the need for the /workspace directory itself, so the build-time
mkdir -p /workspace && chown 1002:0 /workspace && chmod g+rwX /workspacestep (added to fix a permission error when creating it) is now dead code
and is removed along with it.
Verified with: eval-containers run swe-bench --agent swe-agent --model
gcp/gemini-3.5-flash-lite --local --task-id astropy__astropy-12907 —
agent edits land in /testbed and grade.sh's diff against the real bug
now reflects the agent's actual patch (reward:1, passed:true).
Checked against: .agents/benchmarks/RULES.md
Signed-off-by: Doron Chen cdoron@il.ibm.com