SimpleQA
SimpleQA: Measuring short-form factuality in large language models
4,326 个简短的事实型问题,每题有唯一无可争议的答案,针对 GPT-4o 对抗式收集并经两名独立标注者验证,涵盖科学、历史、艺术、地理等。评分模型将每个回答标记为正确、错误或未尝试;主指标为正确率,同时报告 F 分数和校准。它衡量模型是否知道自己知道什么:易产生幻觉的模型即便其他能力出色,得分也会很低。
- 发布
- 2024-11
- 维护者
- OpenAI (Wei et al.)
- 状态
- active
- 污染风险
- high
- 指标
- accuracy (correct) (percent, ↑)
- 题量
- 4,326
- 领域
- factuality knowledge
- 人类
- 无实测基线
备注. Answers are static facts, so retrieval or web search makes the test trivial; only no-tools runs are comparable. Google's SimpleQA Verified (1,000 relabeled questions, 2025) is a separate, stricter variant and its scores should not be mixed with the original. OpenAI stopped reporting SimpleQA in simple-evals in July 2025.
完整账本
| 系统 | 开发者 | 分数 | 日期 | 来源 | 条件 |
|---|---|---|---|---|---|
| gpt-4.5-preview-2025-02-27 | OpenAI | 62.5% | 厂商自报 | tools: no simple-evals README benchmark table (repo deprecated July 2025). | |
| DeepSeek V4 Pro 0813 | DeepSeek | 57.9% | 聚合站 | benchlm.ai lists this under SimpleQA, but DeepSeek's own model card reports 57.9 as SimpleQA-Verified (Pass@1), a different 1,000-question variant; treat as not comparable with the rows above. | |
| o3 | OpenAI | 49.4% | 厂商自报 | tools: no simple-evals README; o3-high 48.6, o4-mini 20.2. | |
| GPT-4o | OpenAI | 38.2% | 论文 | tools: no Table 3 of the SimpleQA paper: percent correct; o1-preview scored 42.7. |