BIG-Bench Hard
BIG-Bench Hard (BBH): 23 challenging BIG-Bench tasks
A suite of 23 BIG-Bench tasks (6,511 examples) on which 2022 language models scored below the average human rater: multi-step arithmetic, logical deduction, tracking shuffled objects, causal judgement, date understanding and others. Scored as exact-match accuracy averaged over tasks, usually 3-shot with chain-of-thought exemplars. It showed that CoT prompting unlocks emergent gains and became a standard reasoning check; frontier models now exceed the best human rater on most tasks.
- Released
- 2022-10
- Maintainer
- Google Research / Stanford (Suzgun et al.)
- Status
- saturated
- Contamination
- high
- Metric
- accuracy (percent, ↑)
- Tasks
- 6,511
- Domains
- reasoning
- human
- 67.7% average BIG-Bench human rater (max rater: 94.4%)
Notes. Answer-only vs CoT prompting changes scores by 10-20 points; the Open LLM Leaderboard v2 reported a 3-shot answer-only variant, so numbers across harnesses are not comparable.
Full ledger
| System | Developer | Score | Date | Source | Conditions |
|---|---|---|---|---|---|
| DeepSeek-V4-Pro-Base | DeepSeek | 87.5% | self-reported | shots: 3 DeepSeek-V4 model card, base-model table (HF repo created 2026-04-22). Base model, exact match, 3-shot; V4-Flash-Base 86.9, V3.2-Base 87.6. | |
| Codex (code-davinci-002), CoT | OpenAI (evaluated by Suzgun et al.) | 73.9% | paper | shots: 3 Table 2 of the BBH paper, all 23 tasks with chain-of-thought prompting (answer-only 56.6; PaLM 540B CoT 65.2; average human rater 67.7, max 94.4). |