MATH
MATH: Measuring Mathematical Problem Solving
12,500 competition mathematics problems (7,500 train, 5,000 test) from AMC 10/12, AIME and similar contests across seven subjects and five difficulty levels, each with a full LaTeX step-by-step solution. Scored as exact match of the boxed final answer after normalization. It was the primary hard-math benchmark until reasoning models pushed it past 95%; the hardest Level 5 subset and the 500-problem MATH-500 split remain in use.
paper · website · leaderboard · dataset · code
- Released
- 2021-03
- Maintainer
- Dan Hendrycks et al. (UC Berkeley)
- Status
- saturated
- Contamination
- high
- Metric
- accuracy (percent, ↑)
- Tasks
- 5,000
- Domains
- math reasoning
- human
- 90% three-time IMO gold medalist (single participant; a CS PhD student scored about 40%)
Notes. Answer-equivalence checking differs across harnesses (string match vs sympy vs LLM grader) and can move scores by several points. The human numbers in the paper are anecdotal single-person measurements.
Full ledger
| System | Developer | Score | Date | Source | Conditions |
|---|---|---|---|---|---|
| o3-high | OpenAI | 98.1% | self-reported | split: MATH-500 shots: 0 simple-evals README; o4-mini-high 98.2 on the same table. Epoch's Level-5 run of gpt-5 (high) reached 98.1 in October 2025. | |
| o1-2024-12-17 (medium) | OpenAI | 94.4% | independent | split: Level 5 Epoch AI Benchmarking Hub run of 2025-01-27 on the 1,324 Level-5 test problems (data export). | |
| gpt-4-turbo-2024-04-09 | OpenAI | 73.4% | self-reported | split: MATH-500 shots: 0 simple-evals README; the MATH column is the 500-problem subset, zero-shot CoT. | |
| GPT-3 175B (few-shot) | OpenAI | 5.2% | paper | Table 2 of the MATH paper, overall accuracy (175B*, few-shot). The paper's human anchors: CS PhD student about 40%, three-time IMO gold medalist 90%. |