SimpleQA
SimpleQA: Measuring short-form factuality in large language models
4,326 short fact-seeking questions with a single indisputable answer, adversarially collected against GPT-4o and verified by two independent annotators, spanning science, history, art, geography and more. A grader model marks each answer correct, incorrect or not attempted; the headline metric is percent correct, with F-score and calibration also reported. It measures whether models know what they know: hallucination-prone models score low even when otherwise capable.
- Released
- 2024-11
- Maintainer
- OpenAI (Wei et al.)
- Status
- active
- Contamination
- high
- Metric
- accuracy (correct) (percent, ↑)
- Tasks
- 4,326
- Domains
- factuality knowledge
- human
- no measured baseline
Notes. Answers are static facts, so retrieval or web search makes the test trivial; only no-tools runs are comparable. Google's SimpleQA Verified (1,000 relabeled questions, 2025) is a separate, stricter variant and its scores should not be mixed with the original. OpenAI stopped reporting SimpleQA in simple-evals in July 2025.
Full ledger
| System | Developer | Score | Date | Source | Conditions |
|---|---|---|---|---|---|
| gpt-4.5-preview-2025-02-27 | OpenAI | 62.5% | self-reported | tools: no simple-evals README benchmark table (repo deprecated July 2025). | |
| DeepSeek V4 Pro 0813 | DeepSeek | 57.9% | aggregator | benchlm.ai lists this under SimpleQA, but DeepSeek's own model card reports 57.9 as SimpleQA-Verified (Pass@1), a different 1,000-question variant; treat as not comparable with the rows above. | |
| o3 | OpenAI | 49.4% | self-reported | tools: no simple-evals README; o3-high 48.6, o4-mini 20.2. | |
| GPT-4o | OpenAI | 38.2% | paper | tools: no Table 3 of the SimpleQA paper: percent correct; o1-preview scored 42.7. |