SWE-bench
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
2,294 个任务实例,基于 12 个流行 Python 代码库中的真实 GitHub issue 及其对应的修复 pull request 构建。给定 issue 文本和代码库快照,智能体需生成补丁;通过运行该 PR 的 FAIL_TO_PASS 和 PASS_TO_PASS 单元测试评分。是最初的智能体编程基准,也是 Verified、Lite、Multimodal、Multilingual 和 Pro 等后续版本的模板。
- 发布
- 2023-10
- 维护者
- Princeton NLP / SWE-bench team (Jimenez et al.)
- 状态
- saturated
- 污染风险
- high
- 指标
- % resolved (percent, ↑)
- 题量
- 2,294
- 领域
- software-engineering code tool-use
- 人类
- 无实测基线
备注. The full 2,294-instance set contains under-specified and untestable tasks (the reason OpenAI and the maintainers curated SWE-bench Verified). Almost all frontier evaluation moved to Verified after August 2024; the Full leaderboard still accepts submissions but receives few, so treat it as historical.
完整账本
| 系统 | 开发者 | 分数 | 日期 | 来源 | 条件 |
|---|---|---|---|---|---|
| Sonar Foundation Agent + Claude 4.5 Opus | Anthropic | 52.62% | 官方榜单 | split: full scaffold: Sonar Foundation Agent Top of the Full leaderboard; submission not run or checked by the SWE-bench team. | |
| SWE-agent 1.0 (Claude 3.7 Sonnet) | Anthropic | 33.83% | 官方榜单 | split: full scaffold: SWE-agent 1.0 | |
| OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022) | Anthropic | 29.38% | 官方榜单 | split: full scaffold: OpenHands CodeAct v2.1 | |
| SWE-agent + GPT 4 (1106) | OpenAI | 12.47% | 官方榜单 | split: full scaffold: SWE-agent | |
| RAG + Claude 2 | Anthropic | 1.96% | 官方榜单 | split: full scaffold: RAG baseline Paper baseline (BM25 retrieval, no agent), checked by the SWE-bench team. |