Currently, we need to edit SWEBench enable to evaluate ourself.
Open swebench/harness/docker_build.py, found this:
eval_commands += [
reset_tests_command,
apply_test_patch_command,
f": '{START_TEST_OUTPUT}'",
test_command,
f": '{END_TEST_OUTPUT}'",
reset_tests_command, # Revert tests after done, leave the repo in the same state as before
]
return eval_commands
Edit into:
eval_commands += [
reset_tests_command,
apply_test_patch_command,
f"echo '{START_TEST_OUTPUT}'",
test_command,
f"echo '{END_TEST_OUTPUT}'",
reset_tests_command, # Revert tests after done, leave the repo in the same state as before
]
return eval_commands
Reinstall swebench.
We need to find a way not to edit source code of swebench
Currently, we need to edit SWEBench enable to evaluate ourself.
Open
swebench/harness/docker_build.py, found this:Edit into:
Reinstall swebench.
We need to find a way not to edit source code of swebench