tau-bench
tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Simulated customer-service conversations in retail (115 tasks) and airline (50 tasks) domains where the agent must talk to an LLM-simulated user, call database APIs and follow a written policy. A task passes only if the final database state matches the annotated goal; pass^k measures whether the same task succeeds in all k independent trials, exposing agent inconsistency. The reference benchmark for policy-following tool agents.
paper · website · leaderboard · dataset · code
- Released
- 2024-06
- Maintainer
- Sierra Research (Yao et al.)
- Status
- retired
- Contamination
- high
- Metric
- pass^1 (percent, ↑)
- Tasks
- 165
- Domains
- tool-use instruction-following general-assistant
- human
- no measured baseline
Notes. The repository README now warns that its airline and retail tasks are outdated and directs users to the tau2-bench repository (tau^3-bench) for fixed tasks; developer reports of 'tau-bench' after mid-2025 usually mean tau2-bench. Scores depend on the user-simulator model, which is not standardised.
Full ledger
| System | Developer | Score | Date | Source | Conditions |
|---|---|---|---|---|---|
| TC (claude-3-5-sonnet-20241022), retail | Anthropic | 69.2% | official | split: retail scaffold: function calling (TC) Repository README leaderboard; date is the model's release date (README gives none). | |
| gpt-4o (function calling), tau-retail | OpenAI | 61.2% | paper | split: retail scaffold: function calling (TC) Table 2 of the paper; user simulated by gpt-4. | |
| TC (claude-3-5-sonnet-20241022), airline | Anthropic | 46% | official | split: airline scaffold: function calling (TC) Repository README leaderboard; date is the model's release date (README gives none). | |
| gpt-4o (function calling), tau-airline | OpenAI | 35.2% | paper | split: airline scaffold: function calling (TC) Table 2 of the paper; user simulated by gpt-4. |