MBPP
Mostly Basic Programming Problems
974 道面向入门程序员的众包短篇 Python 编程任务,每题附自然语言描述、参考解答和三个测试用例;另有 427 题经人工核验的"sanitized"子集。通过对测试执行模型程序评分(pass@1 或解出比例)。与 HumanEval 并列为两个经典早期代码合成基准之一;现已饱和且大量受污染。
- 发布
- 2021-08
- 维护者
- Google Research (Austin et al.)
- 状态
- saturated
- 污染风险
- high
- 指标
- pass@1 (percent, ↑)
- 题量
- 974
- 领域
- code
- 人类
- 无实测基线
备注. Reported numbers mix the full set, the 500-problem test split, the sanitized subset and EvalPlus MBPP+, so compare only like with like.
完整账本
| 系统 | 开发者 | 分数 | 日期 | 来源 | 条件 |
|---|---|---|---|---|---|
| Llama 3.1 405B Instruct | Meta | 88.6% | 厂商自报 | pass_k: 1 shots: 0 Llama 3.1 model card row 'MBPP ++ base version' (EvalPlus MBPP+ base tests, 378 problems): 88.6 for 405B, 86.0 for 70B. Not the original 500-problem test split. Date is the Llama 3.1 release. | |
| 137B LM (few-shot) | Google Research | 59.6% | 论文 | Paper abstract: the largest 137B model synthesizes solutions to 59.6% of MBPP problems with few-shot prompting; fine-tuning adds about 10 points. |