PaperBench
PaperBench: Evaluating AI's Ability to Replicate AI Research
Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch: understand the paper, write a codebase and run the experiments, in a container with GPU access. Grading uses author-co-developed hierarchical rubrics (8,316 gradable leaf nodes) applied by an LLM judge, giving a replication score from 0 to 100. The Code-Dev variant grades code only. It measures long-horizon research engineering, with a measured human (ML PhD) baseline for comparison.
paper · website · leaderboard · dataset · code
- Released
- 2025-04
- Maintainer
- OpenAI (Starace et al.)
- Status
- active
- Contamination
- medium
- Metric
- replication score (percent, ↑)
- Tasks
- 20
- Domains
- research ml-engineering code tool-use
- human
- no measured baseline
Notes. The paper reports that top ML PhDs outperformed models on a 3-paper subset over 48 hours, but the figure is on a subset and not comparable to the full 20-paper score, so no human_baseline is recorded. The repository leaderboard has not been updated since the April 2025 launch; papers and rubrics are public.
Full ledger
| System | Developer | Score | Date | Source | Conditions |
|---|---|---|---|---|---|
| IterativeAgent o1-high | OpenAI | 43.4% | official | split: code-dev scaffold: IterativeAgent reasoning_effort: high PaperBench Code-Dev leaderboard, 3 runs, +/-0.8. | |
| IterativeAgent o1-high (36h limit) | OpenAI | 26% | official | split: full scaffold: IterativeAgent reasoning_effort: high Top of the repository leaderboard; 3 runs, +/-0.3, 36-hour limit. | |
| IterativeAgent o1-high (24h limit) | OpenAI | 24.4% | official | split: full scaffold: IterativeAgent reasoning_effort: high Repository leaderboard, 3 runs, +/-0.7, 24-hour limit. | |
| BasicAgent claude-3.5-sonnet | Anthropic | 21% | official | split: full scaffold: BasicAgent Repository leaderboard, 3 runs, +/-0.8. |