HumanEval
HumanEval: hand-written Python programming problems (Codex paper)
164 道手工编写的 Python 编程题,每题为函数签名加文档字符串,通过运行隐藏单元测试评分,以 pass@k(通常为 pass@1)报告功能正确性。它引入了基于执行的代码评估,是 2021 至 2024 年的标准代码基准;前沿模型如今几乎能解出所有题目,且该题集已广泛存在于训练数据中。
- 发布
- 2021-07
- 维护者
- OpenAI
- 状态
- saturated
- 污染风险
- high
- 指标
- pass@1 (percent, ↑)
- 题量
- 164
- 领域
- code
- 人类
- 无实测基线
备注. Many papers report EvalPlus (HumanEval+) numbers with extra tests, which are lower and not directly comparable. OpenAI's simple-evals stopped reporting new models in July 2025.
完整账本
| 系统 | 开发者 | 分数 | 日期 | 来源 | 条件 |
|---|---|---|---|---|---|
| o4-mini-high | OpenAI | 99.3% | 厂商自报 | pass_k: 1 shots: 0 simple-evals README; effectively the ceiling (163 of 164). | |
| gpt-4-turbo-2024-04-09 | OpenAI | 88.2% | 厂商自报 | pass_k: 1 shots: 0 simple-evals README. | |
| DeepSeek-V4-Pro-Base | DeepSeek | 76.8% | 厂商自报 | pass_k: 1 shots: 0 DeepSeek-V4 model card, base-model table (HF repo created 2026-04-22). Base (pre-trained, not instruction-tuned) model, so lower than chat models above. | |
| Codex 12B | OpenAI | 28.8% | 论文 | pass_k: 1 Codex paper abstract; Codex-S 37.7, GPT-J 11.4, GPT-3 0. 70.2% with 100 samples per problem. |