tau-bench
tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
零售(115 个任务)和航空(50 个任务)领域的模拟客服对话,智能体需与 LLM 模拟的用户交谈、调用数据库 API 并遵守书面政策。仅当最终数据库状态与标注目标一致时任务才算通过;pass^k 衡量同一任务在全部 k 次独立试验中是否都成功,从而暴露智能体的不一致性。是遵循政策的工具智能体的参考基准。
- 发布
- 2024-06
- 维护者
- Sierra Research (Yao et al.)
- 状态
- retired
- 污染风险
- high
- 指标
- pass^1 (percent, ↑)
- 题量
- 165
- 领域
- tool-use instruction-following general-assistant
- 人类
- 无实测基线
备注. The repository README now warns that its airline and retail tasks are outdated and directs users to the tau2-bench repository (tau^3-bench) for fixed tasks; developer reports of 'tau-bench' after mid-2025 usually mean tau2-bench. Scores depend on the user-simulator model, which is not standardised.
完整账本
| 系统 | 开发者 | 分数 | 日期 | 来源 | 条件 |
|---|---|---|---|---|---|
| TC (claude-3-5-sonnet-20241022), retail | Anthropic | 69.2% | 官方榜单 | split: retail scaffold: function calling (TC) Repository README leaderboard; date is the model's release date (README gives none). | |
| gpt-4o (function calling), tau-retail | OpenAI | 61.2% | 论文 | split: retail scaffold: function calling (TC) Table 2 of the paper; user simulated by gpt-4. | |
| TC (claude-3-5-sonnet-20241022), airline | Anthropic | 46% | 官方榜单 | split: airline scaffold: function calling (TC) Repository README leaderboard; date is the model's release date (README gives none). | |
| gpt-4o (function calling), tau-airline | OpenAI | 35.2% | 论文 | split: airline scaffold: function calling (TC) Table 2 of the paper; user simulated by gpt-4. |