Skip to content

fix(swe-agent): point sweagent at /testbed instead of a fake /workspace/repo - #323

Open
cdoron wants to merge 1 commit into
Exgentic:mainfrom
cdoron:pr/swe-agent-repo-name-testbed
Open

fix(swe-agent): point sweagent at /testbed instead of a fake /workspace/repo#323
cdoron wants to merge 1 commit into
Exgentic:mainfrom
cdoron:pr/swe-agent-repo-name-testbed

Conversation

@cdoron

@cdoron cdoron commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

The benchmark's actual checkout — at the buggy commit grade.sh diffs
against — lives at /testbed, not /workspace. The old run.sh instead
git-init'd an empty placeholder repo at /workspace/repo with a single
empty commit, so SWE-agent could never have graded correctly: any patch
it produced was a diff against nothing, not against the real bug.

Switch --env.repo.repo_name to testbed (LocalRepoConfig's preexisting
type is a no-op copy, so it can point directly at the existing checkout)
and drop the /workspace/repo git-init scaffolding entirely. This also
removes the need for the /workspace directory itself, so the build-time
mkdir -p /workspace && chown 1002:0 /workspace && chmod g+rwX /workspace
step (added to fix a permission error when creating it) is now dead code
and is removed along with it.

Verified with: eval-containers run swe-bench --agent swe-agent --model
gcp/gemini-3.5-flash-lite --local --task-id astropy__astropy-12907 —
agent edits land in /testbed and grade.sh's diff against the real bug
now reflects the agent's actual patch (reward:1, passed:true).

Checked against: .agents/benchmarks/RULES.md

Signed-off-by: Doron Chen cdoron@il.ibm.com

…ce/repo

The benchmark's actual checkout — at the buggy commit grade.sh diffs
against — lives at /testbed, not /workspace. The old run.sh instead
git-init'd an empty placeholder repo at /workspace/repo with a single
empty commit, so SWE-agent could never have graded correctly: any patch
it produced was a diff against nothing, not against the real bug.

Switch --env.repo.repo_name to `testbed` (LocalRepoConfig's `preexisting`
type is a no-op copy, so it can point directly at the existing checkout)
and drop the /workspace/repo git-init scaffolding entirely. This also
removes the need for the /workspace directory itself, so the build-time
`mkdir -p /workspace && chown 1002:0 /workspace && chmod g+rwX /workspace`
step (added to fix a permission error when creating it) is now dead code
and is removed along with it.

Verified with: eval-containers run swe-bench --agent swe-agent --model
gcp/gemini-3.5-flash-lite --local --task-id astropy__astropy-12907 —
agent edits land in /testbed and grade.sh's diff against the real bug
now reflects the agent's actual patch (reward:1, passed:true).

Checked against: .agents/benchmarks/RULES.md

Signed-off-by: Doron Chen <cdoron@il.ibm.com>

@elronbandel elronbandel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified against grade.sh directly: it does cd /testbed; git diff — that's the real grading target, and it's what TASK already told the agent ("the repository at /testbed"). Before this fix sweagent operated on /workspace/repo, an empty git-init'd placeholder totally disconnected from /testbed — meaning every swe-agent/swe-bench run scored 0 unconditionally, and the TASK text contradicted the actual environment. This fix makes repo_name, safe.directory, and TASK all agree on /testbed. Confirmed no stale /workspace references remain in the file. This also fully obsoletes #318.

@elronbandel elronbandel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Walking back my earlier approval — the underlying bug and grading analysis were correct, but the fix is scoped too narrowly and breaks other benchmarks.

`containers/agents/swe-agent/Dockerfile` is the generic agent image, shared across every benchmark paired with swe-agent (agents/RULES.md rule 4: "the agent MUST NOT know which benchmark it is running in"). This PR hardcodes `--env.repo.repo_name=testbed` unconditionally. Checked the other benchmarks with existing swe-agent replay fixtures — `apps`, `code-contests`, `math-500`, `mmmu` — none of them have a `/testbed` directory (code-contests/math-500 use `/app`, apps/mmmu declare no WORKDIR at all). sweagent's `preexisting` repo type assumes the path already exists (no copy, no init) — pointed at a nonexistent `/testbed` for those four, this will very likely fail at startup, breaking `apps-2999-swe-agent`, `code-contests-98-swe-agent`, `math-500-299-swe-agent`, and `mmmu-359-swe-agent` replay tests.

The swe-bench-specific fix (pointing at /testbed) is correct for swe-bench, but needs to stay benchmark-agnostic — e.g. detect the actual cwd/WORKDIR at runtime the way #308 did for openclaw (`$(pwd)` instead of a hardcoded path), rather than hardcoding `testbed` into the shared agent image.

Separately (not this PR's fault): this now has a real merge conflict with main since #319 already landed independently and touches the same block.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants