SWE-bench
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
2,294 task instances built from real GitHub issues and their resolving pull requests across 12 popular Python repositories. Given the issue text and a repository snapshot, an agent must produce a patch; it is scored by running the PR's FAIL_TO_PASS and PASS_TO_PASS unit tests. The original agentic-coding benchmark and the template for the Verified, Lite, Multimodal, Multilingual and Pro descendants.
paper · website · leaderboard · dataset · code
- Released
- 2023-10
- Maintainer
- Princeton NLP / SWE-bench team (Jimenez et al.)
- Status
- saturated
- Contamination
- high
- Metric
- % resolved (percent, ↑)
- Tasks
- 2,294
- Domains
- software-engineering code tool-use
- human
- no measured baseline
Notes. The full 2,294-instance set contains under-specified and untestable tasks (the reason OpenAI and the maintainers curated SWE-bench Verified). Almost all frontier evaluation moved to Verified after August 2024; the Full leaderboard still accepts submissions but receives few, so treat it as historical.
Full ledger
| System | Developer | Score | Date | Source | Conditions |
|---|---|---|---|---|---|
| Sonar Foundation Agent + Claude 4.5 Opus | Anthropic | 52.62% | official | split: full scaffold: Sonar Foundation Agent Top of the Full leaderboard; submission not run or checked by the SWE-bench team. | |
| SWE-agent 1.0 (Claude 3.7 Sonnet) | Anthropic | 33.83% | official | split: full scaffold: SWE-agent 1.0 | |
| OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022) | Anthropic | 29.38% | official | split: full scaffold: OpenHands CodeAct v2.1 | |
| SWE-agent + GPT 4 (1106) | OpenAI | 12.47% | official | split: full scaffold: SWE-agent | |
| RAG + Claude 2 | Anthropic | 1.96% | official | split: full scaffold: RAG baseline Paper baseline (BM25 retrieval, no agent), checked by the SWE-bench team. |