Skip to content

fix(benchmark): remove personal SWE-bench path defaults - #739

Open
beruro wants to merge 3 commits into
developfrom
junyu/fix-benchmark-personal-paths
Open

fix(benchmark): remove personal SWE-bench path defaults#739
beruro wants to merge 3 commits into
developfrom
junyu/fix-benchmark-personal-paths

Conversation

@beruro

@beruro beruro commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

The SWE-bench Pro harness default embedded one developer's absolute macOS path. It was invalid on other machines, leaked a personal directory layout, and delayed the configuration error until filesystem access.

Solution

  • Remove the personal filesystem default.
  • Centralize resolution in benchmark/paths.rs with explicit precedence: caller-provided value, then ORGII_SWE_BENCH_PRO_REPO_PATH, then a typed not-configured error.
  • Treat blank values as unset and trim resolved values.
  • Surface the same actionable configuration error during preflight and local-Docker plan creation.
  • Pass already-resolved evaluator/run-script paths into command preview instead of resolving them again.
  • Cover precedence, blank handling, trimming, and error messaging with pure unit tests.

The branch is merged with the current develop; GitHub reports it mergeable.

Potential risks

  • The original developer machine must now set ORGII_SWE_BENCH_PRO_REPO_PATH instead of relying on the removed implicit path. This is the intended compatibility break.
  • There is not yet an in-app setting for this path; the environment variable remains the only production configuration source.
  • In local_docker mode, missing configuration now fails during plan creation instead of returning a plan that fails later. The existing frontend error path handles the returned message; no DTO or wire shape changed.
  • Rollback is a normal revert and restores the unsafe personal-path default. No data, schema, dependency, or persistence migration is involved.

Verification

  • cargo test -p org2 benchmark::pathsPASS, 7 passed / 0 failed.
  • rg -n "/Users/laptop-h|/Users/junyu" src-tauri/src/benchmark/PASS, no personal path remains.
  • git diff --checkPASS.
  • Merge of current origin/developPASS, no conflicts; targeted tests were run after integration.

@beruro
beruro marked this pull request as ready for review August 8, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant