HumanEval
HumanEval: hand-written Python programming problems (Codex paper)
164 hand-written Python programming problems, each a function signature plus docstring, scored by running hidden unit tests and reporting pass@k (usually pass@1) for functional correctness. It introduced execution-based code evaluation and was the standard code benchmark from 2021 to 2024; frontier models now solve nearly every problem and the set is widely present in training data.
- Released
- 2021-07
- Maintainer
- OpenAI
- Status
- saturated
- Contamination
- high
- Metric
- pass@1 (percent, ↑)
- Tasks
- 164
- Domains
- code
- human
- no measured baseline
Notes. Many papers report EvalPlus (HumanEval+) numbers with extra tests, which are lower and not directly comparable. OpenAI's simple-evals stopped reporting new models in July 2025.
Full ledger
| System | Developer | Score | Date | Source | Conditions |
|---|---|---|---|---|---|
| o4-mini-high | OpenAI | 99.3% | self-reported | pass_k: 1 shots: 0 simple-evals README; effectively the ceiling (163 of 164). | |
| gpt-4-turbo-2024-04-09 | OpenAI | 88.2% | self-reported | pass_k: 1 shots: 0 simple-evals README. | |
| DeepSeek-V4-Pro-Base | DeepSeek | 76.8% | self-reported | pass_k: 1 shots: 0 DeepSeek-V4 model card, base-model table (HF repo created 2026-04-22). Base (pre-trained, not instruction-tuned) model, so lower than chat models above. | |
| Codex 12B | OpenAI | 28.8% | paper | pass_k: 1 Codex paper abstract; Codex-S 37.7, GPT-J 11.4, GPT-3 0. 70.2% with 100 samples per problem. |