DeepSearchQA
DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
900 hand-crafted, time-anchored web-research prompts across 17 fields whose answers are exhaustive sets rather than single facts (65% set answers). Each prompt is a causal chain of dependent look-ups, so agents must plan multi-step searches, collate fragments from many sources, de-duplicate entities and decide when to stop. Answers are judged for semantic set match by a fixed Gemini 2.5 Flash autorater; the primary metric is mean per-prompt F1, with fully-correct and fully-incorrect rates as diagnostics. It targets the comprehensiveness gap that single-answer benchmarks like BrowseComp miss.
paper · website · leaderboard · dataset · code
- Released
- 2025-12
- Maintainer
- Google DeepMind (Gupta, Chatterjee, Haas, Tao et al.); leaderboard run by Kaggle
- Status
- active
- Contamination
- medium
- Metric
- F1 (mean over prompts) (percent, ↑)
- Tasks
- 900
- Domains
- web research factuality tool-use
- human
- no measured baseline
Notes. Announced 2025-12-11 with the Gemini Deep Research agent; arXiv paper followed 2026-01-28. The official Kaggle leaderboard (last updated 2025-12-11, 13 systems) is independently run by Kaggle and reports F1 with 95% CIs; the dataset card warns that a different autorater or grading prompt gives statistically different scores, so vendor-reported numbers (Meta, Anthropic, Moonshot via aggregators) may not be comparable. Prompts and gold answers are public on Hugging Face/Kaggle, and ground truth can drift as web sources change.
Full ledger
| System | Developer | Score | Date | Source | Conditions |
|---|---|---|---|---|---|
| Claude Opus 5 | Anthropic | 95% | aggregator | tools Vendor-published number republished by aggregator (tied with Kimi K3 at 95.0); harness and autorater unspecified, so not comparable to the Kaggle-run board. | |
| Gemini Deep Research Agent | 81.9% | official | tools Kaggle-run leaderboard, rank 1; 66.1% fully correct, 10.0% fully incorrect, 10.3% correct with excessive answers. Leaderboard last updated 2025-12-11. | ||
| GPT-5 Pro | OpenAI | 79% | official | tools Kaggle-run leaderboard, rank 2; 65.2% fully correct, 14.1% fully incorrect. | |
| GPT-5.4 | OpenAI | 78.6% | official | tools Kaggle-run leaderboard, rank 3; 63.7% fully correct. Date is the leaderboard's 'last updated' stamp. | |
| Claude Sonnet 4.6 | Anthropic | 56.5% | official | tools Kaggle-run leaderboard, rank 6; 50.8% fully correct, 41.1% fully incorrect. Date is the leaderboard's 'last updated' stamp. |