MBPP

Mostly Basic Programming Problems

974 crowd-sourced short Python programming tasks aimed at entry-level programmers, each with a natural-language description, a reference solution and three test cases; a 427-problem hand-verified 'sanitized' subset is also used. Scored by executing the model's program against the tests (pass@1 or fraction solved). One of the two canonical early code-synthesis benchmarks alongside HumanEval; now saturated and largely contaminated.

paper · website · dataset · code

88.6%
Llama 3.1 405B Instruct
self-reported 2024-07-23

Released: 2021-08
Maintainer: Google Research (Austin et al.)
Status: saturated
Contamination: high
Metric: pass@1 (percent, ↑)
Tasks: 974
Domains: code
human: no measured baseline

Notes. Reported numbers mix the full set, the 500-problem test split, the sanitized subset and EvalPlus MBPP+, so compare only like with like.

Full ledger

System	Developer	Score	Date	Source	Conditions
Llama 3.1 405B Instruct	Meta	88.6%	2024-07-23	self-reported	pass_k: 1 shots: 0 Llama 3.1 model card row 'MBPP ++ base version' (EvalPlus MBPP+ base tests, 378 problems): 88.6 for 405B, 86.0 for 70B. Not the original 500-problem test split. Date is the Llama 3.1 release.
137B LM (few-shot)	Google Research	59.6%	2021-08-16	paper	Paper abstract: the largest 137B model synthesizes solutions to 59.6% of MBPP problems with few-shot prompting; fine-tuning adds about 10 points.