LLM Benchmarks Tracker

17,900 execution-verified CadQuery programs across 106 industrial part families (gears, springs, drills, fittings), about half anchored to real ISO/DIN/EN/ASME/IEC specification tables. The prime task, Vision2Code, shows four orthographic renders and asks for a CadQuery program that is re-executed and scored by IoU-score (voxel IoU times execution rate); matched Vision QA, Code QA and Code Edit tasks isolate perception, parametric abstraction and program synthesis. An agentic variant adds a Python sandbox to render, measure and iterate. It is the yardstick vendors cite for AI-for-hardware.

6,000 short-answer factual questions generated from authoritative academic and industry sources across 42 topics in six domains (business, humanities, health, law, software engineering, science and engineering). An LLM grader labels each answer correct, partial, incorrect or not attempted; the Omniscience Index (-100 to 100) awards correct answers, subtracts incorrect ones and leaves abstentions neutral, so it measures knowledge calibration rather than raw recall. Accuracy and hallucination rate are reported separately and feed the Artificial Analysis Intelligence Index.

Real work deliverables (legal briefs, spreadsheets, slide decks, CAD, schedules, video edits) drawn from 44 occupations in the 9 US sectors that contribute most to GDP, written and reviewed by professionals averaging 14 years of experience. A model receives the request plus reference files and produces the deliverable in one shot; occupational experts blindly compare it with the human expert's work and the score is the share of comparisons rated better than or as good as the expert. The main public yardstick for economically valuable knowledge work.

5,000 realistic multi-turn, multilingual health conversations between a model and a layperson or clinician, spanning seven themes such as emergency referrals, context seeking and global health. Each conversation carries a physician-written rubric (48,562 criteria from 262 physicians in 60 countries); a GPT-4.1 grader checks which criteria the final response meets and the score is points earned over the maximum. Hard (1,000) and Consensus (3,671) subsets isolate unsaturated and physician-validated criteria. The reference evaluation for medical helpfulness and safety.

Second-generation ARC grid puzzles designed to defeat brute-force search and test symbolic interpretation, compositional reasoning and context-dependent rule application. 1,000 public training tasks plus calibrated 120-task public, semi-private and private evaluation sets; every task was solved by at least two humans within two attempts. Scored as the percentage of test outputs exactly correct within two attempts, always reported together with cost per task. It launched with ARC Prize 2025 as the successor to ARC-AGI-1.

1,581 GUI grounding instructions, each on a unique authentic high-resolution screenshot from 23 professional applications (CAD, IDEs, creative suites, scientific tools, office) and three operating systems. The model must output the click point or box for the element an instruction refers to; targets average 0.07% of the screen, and accuracy is the share of predictions landing inside the ground-truth box (micro-averaged over text and icon targets). The standard stress test for the visual grounding that computer-use agents depend on.

2,500 expert-written closed-ended questions (multiple choice and exact-match short answer, about 14% with images) across mathematics, natural sciences, humanities and more, filtered so frontier models failed them at collection time. Scored as accuracy by an LLM judge (o3-mini) against the reference answer; calibration error is also reported. Designed as the final broad academic benchmark, it is now the main frontier knowledge-reasoning test.

225 of the hardest Exercism exercises in C++, Go, Java, JavaScript, Python and Rust, selected because at most 3 of 7 reference models solved them. The model runs inside the aider CLI, must edit the files using a specified edit format, and gets one retry with the failing test output; the score is the percentage of exercises whose unit tests pass after two attempts. A widely quoted measure of instruction-following code editing that also exposes edit-format compliance and cost per run.

4,326 short fact-seeking questions with a single indisputable answer, adversarially collected against GPT-4o and verified by two independent annotators, spanning science, history, art, geography and more. A grader model marks each answer correct, incorrect or not attempted; the headline metric is percent correct, with F-score and calibration also reported. It measures whether models know what they know: hallucination-prone models score low even when otherwise capable.

Hundreds of original, unpublished research-grade mathematics problems written and vetted by expert mathematicians, from hard undergraduate (Tier 1) through advanced graduate (Tier 3) to research-level Tier 4, each with an automatically verifiable answer. Models may reason and run Python within a token budget and submit an answer function; scored as the fraction of problems solved. Kept private to avoid contamination, it is the main measure of frontier mathematical reasoning, and Epoch runs all evaluations itself.

1,730 MMMU questions that survived filtering out items answerable by text-only models, each with the option set expanded to ten candidates, plus a vision-only setting in which the question is rendered inside the image so the model must read and see at once. Scored as accuracy; the headline number averages the standard (10-option) and vision settings. It is the current standard for expert-level multimodal reasoning because it removes the text-only shortcuts that inflated MMMU.

About 12,000 questions across 14 disciplines built from MMLU, STEM sites, TheoremQA and SciBench, with the answer set expanded from four to ten options and trivial or noisy items removed. Scored as accuracy with chain-of-thought (5-shot in the official setup). Reduces guessing headroom and prompt sensitivity, making it the standard replacement for MMLU when comparing knowledge and reasoning.

A synthetic long-context suite of 13 tasks in four categories (needle-in-a-haystack retrieval variants, multi-hop variable tracing, aggregation of frequent words, and question answering over long inputs) generated at configurable lengths from 4K to 128K tokens or more. Scored as accuracy averaged across tasks at each length; a model's 'effective context' is the longest length at which it beats a fixed threshold (Llama-2-7B at 4K, 85.6%). It exposes the gap between claimed and usable context windows.

Competitive-programming problems continuously collected from LeetCode, AtCoder and Codeforces contests, each tagged with its release date so that models can be scored only on problems published after their training cutoff. Code generation is scored as pass@1 against hidden tests (averaged over samples); self-repair, code execution and test-output prediction are separate scenarios. The rolling release-date filter makes it the standard contamination-aware coding benchmark, though the window chosen changes the score.

Evaluates how accurately a model produces function (tool) calls from natural-language requests. V1 introduced abstract-syntax-tree matching of single, multiple and parallel calls in Python, Java, JavaScript and REST; V2 added user-contributed live functions; V3 added multi-turn, multi-step and state-tracking scenarios; V4 added web-search and memory agentic tasks and format-sensitivity checks. The overall score is an unweighted average of sub-category accuracies, reported with cost and latency. The de-facto standard for tool-calling ability.

11,500 college-level questions that pair text with images (charts, diagrams, maps, tables, chemical structures, music sheets) across six disciplines and 30 subjects, collected from exams, quizzes and textbooks. Scored as accuracy on the 900-question validation set (the test set of 10,500 is graded via EvalAI). The first broad expert-level multimodal benchmark; it is now close to expert-human accuracy and has been superseded by MMMU-Pro.

541 prompts each containing one or more of 25 types of verifiable instructions, such as 'write at least 400 words', 'respond in JSON', 'include the keyword X three times' or 'no commas'. Compliance is checked programmatically, giving prompt-level and instruction-level accuracy under strict and loose matching. Cheap, reproducible and free of LLM judges, it is the standard check that a chat model actually obeys formatting constraints.

198 four-option multiple-choice questions in biology, physics and chemistry written by domain PhDs and validated so that experts agree while skilled non-experts with unrestricted web access fail. The Diamond subset keeps only questions both experts answered correctly and most non-experts missed. The standard frontier science-reasoning benchmark; noise floor is roughly plus or minus 3 points at n=198.

Live crowdsourced human-preference ranking of chat models. Visitors type any prompt, receive answers from two anonymous models side by side and vote; millions of pairwise votes are fit with a Bradley-Terry model (originally online Elo) to produce an Arena score with bootstrap confidence intervals, plus style-controlled and category views. Because prompts are fresh and judged by real users it is hard to contaminate and captures perceived helpfulness, but it rewards persuasive formatting and depends on which models are sampled.

108 long-horizon computer-use workflows (median human time about 1.6 hours, roughly 300 agent tool calls) spanning research, creative, engineering, finance, administrative and healthcare work across desktop apps and 31 self-hosted websites. Tasks are graded on an average of 27 execution-based checkpoints, yielding a binary completion rate (primary metric, reported at a 500-step budget) and a partial score. Tasks include dynamic mid-task events, hidden state and simulated-user clarification, so it probes the failure modes OSWorld 1.0 no longer separates.

Business-workflow tasks across Sales, Marketing, Operations, Support, Finance and HR in which an agent, given one trigger message, must discover the right REST endpoints (BM25 search over ~500 endpoint schemas from 47 simulated SaaS apps), follow policy documents hidden in the environment, avoid decoy records, and mutate the state of a simulated company. Grading is deterministic end-state assertions, including negative ones; the official score is the strict fraction of tasks with every assertion passing on a private held-out set (600+ tasks); a 600-task public set is released for research.

Interactive, turn-based game-like environments built only from Core Knowledge priors, with no instructions: an agent must explore, infer the goal, build a world model and plan. 25 public demo environments plus 55 semi-private (API-tested) and 55 fully private (competition) environments, each verified fully solvable by untrained humans. Scoring is efficiency-based against human action baselines, so 100% means solving every level as efficiently as humans. The first interactive generation of the ARC-AGI series.

Hard, human-authored tasks that an agent must complete inside a real terminal environment (software engineering, scientific computing, ML, security, data processing). Each task ships its own container, a human solution and tests; the score is the fraction of tasks whose tests pass, averaged over several trials with 95% confidence intervals. The de-facto benchmark for CLI coding agents, run through the Harbor framework.

900 hand-crafted, time-anchored web-research prompts across 17 fields whose answers are exhaustive sets rather than single facts (65% set answers). Each prompt is a causal chain of dependent look-ups, so agents must plan multi-step searches, collate fragments from many sources, de-duplicate entities and decide when to stop. Answers are judged for semantic set match by a fixed Gemini 2.5 Flash autorater; the primary metric is mean per-prompt F1, with fully-correct and fully-incorrect rates as diagnostics. It targets the comprehensiveness gap that single-answer benchmarks like BrowseComp miss.

108 manually sourced tasks that require an agent to drive 32 real software applications through 604 tools (mostly MCP servers) such as Google Calendar, Notion, Canvas, WooCommerce, Kubernetes and BigQuery, starting from realistic seeded environment states and taking roughly 20 turns each. Every task is graded by a dedicated execution-based evaluation script on the resulting state; the leaderboard reports Pass@1 over three runs plus Pass@3 and Pass^3. It is the broadest public test of general MCP-style tool use across heterogeneous apps.

1,865 human-verified, long-horizon software engineering tasks from 41 repositories (public set: 731 tasks from copyleft-licensed OSS repos; commercial set: 276 tasks from proprietary startup codebases; held-out set: 858 tasks). Each task gives an augmented problem statement, requirements and optional interface; the patch is scored by fail-to-pass and pass-to-pass tests. Reference solutions average about 107 changed lines across 4 files, so it measures multi-file, enterprise-style work that SWE-bench Verified no longer separates.

Successor to tau-bench that adds a telecom troubleshooting domain (114 tasks) in which both the agent and the simulated user have tools, so the agent must coordinate and guide the user rather than act alone; the verified retail (115) and airline (50) domains are retained. Tasks are generated compositionally from atomic sub-tasks and scored on final database state; pass^k measures reliability across repeated trials. Now maintained as tau^3-bench with a banking knowledge domain and a voice modality.

886 tasks over on-site images, videos and documents captured in real factories, warehouses and retail stores (711 perception, 121 decision-making, 54 combination tasks), written from interviews with site workers and managers. An agent must extract information, detect safety or procedural violations and produce reports; answers are scored against ground truth with a weighted mix of exact and near-match scoring on a 0-1 scale (reported here as percent). It tests multimodal agents on physical-world field operations rather than digital environments.

Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch: understand the paper, write a codebase and run the experiments, in a container with GPU access. Grading uses author-co-developed hierarchical rubrics (8,316 gradable leaf nodes) applied by an LLM judge, giving a replication score from 0 to 100. The Code-Dev variant grades code only. It measures long-horizon research engineering, with a measured human (ML PhD) baseline for comparison.

1,266 human-written questions whose short, verifiable answers can only be found by persistently browsing many web pages and combining entangled constraints; human trainers verified that other people could not answer them within ten minutes and that then-current models failed. Scored as accuracy against the reference answer via a grader. It isolates deep web-research persistence rather than latent knowledge, so it separates browsing agents sharply.

300 curated issue-resolution tasks from 42 repositories in C, C++, Go, Java, JavaScript/TypeScript, PHP, Ruby and Rust, built with the SWE-bench collection pipeline and scored by fail-to-pass and pass-to-pass tests. It checks whether agentic coding ability transfers beyond Python, where scaffolds and models are heavily tuned to SWE-bench Verified.

75 Kaggle competitions (22 low, 38 medium, 15 high complexity) in which an agent must read the task, prepare data, train models and submit predictions inside a sandbox within a compute budget. The score is the fraction of competitions in which the submission would have earned at least a bronze medal on the real Kaggle leaderboard. The standard test of end-to-end ML engineering rather than isolated coding.

500 real GitHub issues from 12 Python repositories, filtered by human annotators to remove under-specified or untestable tasks. An agent receives the issue text and the repository, must produce a patch, and is scored by hidden FAIL_TO_PASS and PASS_TO_PASS tests. The de-facto standard for agentic coding; scores depend heavily on scaffold and step/cost limits.

369 real computer-use tasks in a live Ubuntu VM spanning Chrome, LibreOffice, GIMP, VLC, VS Code, Thunderbird, OS file operations and multi-app workflows. Agents see screenshots (and optionally the accessibility tree), emit mouse/keyboard actions, and are scored by execution-based checkers on the final machine state. The standard benchmark for GUI computer-use agents; OSWorld-Verified (July 2025) fixed 300+ task issues and is the basis of the current official leaderboard.

910 web tasks on Classifieds, Shopping and Reddit sites that require reading images (product photos, listings, posted pictures) to complete, extending WebArena's functional evaluation with visually grounded checks. Agents receive screenshots (optionally Set-of-Marks annotated) plus the accessibility tree and must act in the browser. It isolates the multimodal-perception gap that text-only web agents cannot close.

466 real-world questions (166 public validation, 300 private test) that need web browsing, file handling, multimodal reading and tool use to reach a short, unambiguous answer; scored by quasi-exact match after normalisation. Three difficulty levels, with Level 3 requiring long tool-use chains. Questions are easy for humans, so the gap to human accuracy is the headline signal for general assistants.

812 long-horizon natural-language tasks on self-hosted, fully functional websites (e-commerce, forum, GitLab, CMS) plus a map, calculator and wiki. Agents act through a browser and are scored by programmatic functional checks on the final site state or answer rather than by matching action sequences. The first reproducible end-to-end web-agent benchmark and still the reference for text/DOM web agents.