Skip to content

Tool system: SWE-bench integration and public benchmark #363

Description

@Delqhi

Problem

SIN-Code has a verification gate but no published SWE-bench score or public benchmark. OpenHands reaches 77.6% on SWE-bench; we have no measured comparison. The verification gate is our moat but unproven externally.

What to do

  1. internal/eval/swebench.go — SWE-bench harness:
    • Load SWE-bench JSON instances
    • Run sin-code on each issue
    • Apply predicted patch
    • Run evaluation with test_patch
    • Record pass/fail
  2. sin-code eval swebench --dataset swebench.json --output results.json
  3. Publish results to docs/swebench.md
  4. CI: n8n-delegated nightly run (M1)
  5. Compare against OpenHands/Codex baselines

Acceptance criteria

  • SWE-bench harness runs
  • Generates results JSON
  • Pass/fail measured
  • Results published
  • CI nightly run

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions