# LLM Benchmarks Tracker

LLM Benchmarks Tracker is a sourced, schema-validated catalogue of 54 LLM and agent evaluation benchmarks (31 model, 23 agent) and 18 evaluators, in which every score records who published it, under which conditions, and what kind of source it came from.

## What it is

A data repository and generated site describing how language models and agents are measured. Its 54 benchmarks split into two layers: 31 model benchmarks (static prompt-and-response scoring) and 23 agent benchmarks (interactive environments where the system acts and is scored on task completion). Each benchmark file records what it tests, task count and splits, the metric, release date, maintainer, a saturation status (active, saturating, saturated, retired), a contamination risk describing how exposed its test set is, a measured human baseline when one exists with a citation, the supersession chain to and from related benchmarks, and links to paper, leaderboard, dataset and code. Alongside them 252 result rows form append-only ledgers — a new score is a new row, never an edit — and 18 evaluator entries describe the frameworks, leaderboards, independent evaluators and aggregators that produce those numbers. Everything is JSON validated against published JSON Schema 2020-12 files, and the site, the JSON API, this file and claims.json are generated from it by scripts/build.py.

### Where the numbers come from

This project runs no evaluations and measures nothing itself: every score is a third-party result republished with its provenance. Across the 252 rows the source kinds are: official-leaderboard 104, paper 64, developer-report 55, independent-evaluation 16, aggregator 13. A row's source URL is the page the number was published on and accessed is the date it was read there; conditions records the split, tools, reasoning effort, scaffold, pass@k, shots and cost per task that the source stated, and omits what it did not. The organisations that do produce numbers are catalogued as evaluators: independent evaluators and aggregators (Artificial Analysis, BenchLM, Epoch AI Benchmarking Hub, Scale SEAL Leaderboards, Vals AI), maintainer-run leaderboards (Holistic Agent Leaderboard (HAL), LMArena, LiveBench, Open LLM Leaderboard, SWE-rebench) and harnesses you run yourself (DeepEval, EvalScope, HELM, Inspect AI, LM Evaluation Harness, Lighteval, OpenAI Evals, OpenCompass).

### How it refreshes

.github/workflows/sync.yml runs scripts/sync_ledgers.py twice a week (07:00 UTC on Monday and Thursday) and appends new top scores from the sources that publish machine-readable results; schema validation, append-only ordering and duplicate detection gate every write, and git history is the audit log. Everything else — vendor posts, aggregator pages, newly released benchmarks — is added by a person in a pull request, and CI rejects schema violations, dangling references, unsourced rows and stale README tables. .github/workflows/pages.yml reruns scripts/build.py and redeploys on every push to main, so the site, the API, llms.txt, llms-full.txt and claims.json are regenerated together from data/ and cannot disagree with each other.

## Install

```bash
git clone https://github.com/alloevil/llm-benchmarks-tracker
cd llm-benchmarks-tracker
pip install -e ".[dev]"
```

Nothing needs installing to read the data:

```bash
curl https://alloevil.github.io/llm-benchmarks-tracker/api/v1/index.json
```

## Quickstart

```bash
python scripts/validate.py          # schema + cross-file invariants
python scripts/build.py             # README tables (en + zh), dist/ site, JSON API, llms.txt, claims.json
pytest                              # validator and build contract tests
```

Using the published data instead of the repository:

```python
import json, urllib.request
api = "https://alloevil.github.io/llm-benchmarks-tracker/api/v1/"
benchmarks = json.load(urllib.request.urlopen(api + "benchmarks.json"))["benchmarks"]
agentic = [b for b in benchmarks if b["status"] == "active" and b["layer"] == "agent"]
for b in agentic:
    s = b["sota"]
    print(b["name"], s and f'{s["value"]} {s["system"]} ({s["source"]["kind"]})')
```

## Verifiable claims

Machine-readable copy: https://alloevil.github.io/llm-benchmarks-tracker/claims.json. Every value below is counted from data/ at build time, so it cannot drift from the repository, and each one carries a check: a command that recomputes the figure from data/ without calling the generator, run by .github/workflows/claims.yml on every push and weekly. None of them is a measurement made by this project; they describe the catalogue and the provenance of the scores in it.

- **54** — The catalogue describes 54 evaluation benchmarks: 31 model benchmarks and 23 agent benchmarks. Metric: benchmark metadata files that pass schema/benchmark.schema.json. Method: scripts/dataset.py::load validates every data/benchmarks/*.json against the published schema and counts what loads. Repro: python3 scripts/validate.py. Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/benchmarks.
- **31/23** — The 54 benchmarks split into 31 model benchmarks and 23 agent benchmarks. Metric: benchmark metadata files by layer, model benchmarks first. Method: layer field per benchmark file; the receipt recounts it straight from data/benchmarks/*.json. Repro: python3 -c "import json, glob;B=[json.load(open(p)) for p in glob.glob('data/benchmarks/*.json')];L=[b['layer'] for b in B];print(L.count('model'), L.count('agent'), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/benchmarks.
- **18** — The catalogue describes 18 evaluators: frameworks, leaderboards, independent evaluators and aggregators. Metric: evaluator metadata files that pass schema/evaluator.schema.json. Method: scripts/dataset.py::load validates every data/evaluators/*.json against the published schema and counts what loads. Repro: python3 scripts/validate.py. Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/evaluators.
- **252/252** — All 252 result rows carry a source URL, a source kind and an access date. Metric: result rows with source.url, source.kind and source.accessed present. Method: schema/results.schema.json marks all three fields required; scripts/validate.py fails the build on a row without them. Repro: python3 -c "import json, glob;R=[r for p in glob.glob('data/results/*.json') for r in json.load(open(p))['results']];print(sum(1 for r in R if r['source'].get('url') and r['source'].get('kind') and r['source'].get('accessed')), len(R), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/blob/main/schema/results.schema.json.
- **2026-09-13** — The newest access date across the 252 result rows is 2026-09-13, which is the date the README publishes as `data as of`. Metric: newest source.accessed date over all result rows. Method: scripts/build.py::data_as_of takes the maximum access date, so the README stamp moves with the data and not with the build date; the receipt recomputes that maximum from data/results/*.json. Repro: python3 -c "import json, glob;R=[r for p in glob.glob('data/results/*.json') for r in json.load(open(p))['results']];print(max(r['source'].get('accessed') or r['date'] for r in R))". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results.
- **52%** — 52% of the current top scores come from an official leaderboard, a paper or an independent evaluation rather than a vendor self-report or an aggregator. Metric: share of per-benchmark top scores whose source.kind is official-leaderboard, paper or independent-evaluation. Method: scripts/build.py::provenance_share over Dataset.sota() of every benchmark; the receipt picks the top row again from each ledger (higher_is_better, ties to the earliest date) and counts the trusted kinds. Repro: python3 -c "import json, glob;S=[(b, json.load(open('data/results/' + b['id'] + '.json'))['results']) for b in [json.load(open(p)) for p in glob.glob('data/benchmarks/*.json')]];S=[(b, r) for b, r in S if r];print(round(100 * sum(1 for b, r in S if min(r, key=lambda x: (-(1 if b['metric']['higher_is_better'] else -1) * x['value'], x['date']))['source']['kind'] in ('official-leaderboard', 'paper', 'independent-evaluation')) / len(S)), '%', sep='')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results.
- **104/252** — 104 of the 252 result rows have source kind official-leaderboard. Metric: result rows whose source.kind is official-leaderboard. Method: count over data/results/*.json after schema validation. Repro: python3 -c "import json, glob;R=[r for p in glob.glob('data/results/*.json') for r in json.load(open(p))['results']];print(sum(r['source']['kind'] == 'official-leaderboard' for r in R), len(R), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results.
- **64/252** — 64 of the 252 result rows have source kind paper. Metric: result rows whose source.kind is paper. Method: count over data/results/*.json after schema validation. Repro: python3 -c "import json, glob;R=[r for p in glob.glob('data/results/*.json') for r in json.load(open(p))['results']];print(sum(r['source']['kind'] == 'paper' for r in R), len(R), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results.
- **55/252** — 55 of the 252 result rows have source kind developer-report. Metric: result rows whose source.kind is developer-report. Method: count over data/results/*.json after schema validation. Repro: python3 -c "import json, glob;R=[r for p in glob.glob('data/results/*.json') for r in json.load(open(p))['results']];print(sum(r['source']['kind'] == 'developer-report' for r in R), len(R), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results.
- **16/252** — 16 of the 252 result rows have source kind independent-evaluation. Metric: result rows whose source.kind is independent-evaluation. Method: count over data/results/*.json after schema validation. Repro: python3 -c "import json, glob;R=[r for p in glob.glob('data/results/*.json') for r in json.load(open(p))['results']];print(sum(r['source']['kind'] == 'independent-evaluation' for r in R), len(R), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results.
- **13/252** — 13 of the 252 result rows have source kind aggregator. Metric: result rows whose source.kind is aggregator. Method: count over data/results/*.json after schema validation. Repro: python3 -c "import json, glob;R=[r for p in glob.glob('data/results/*.json') for r in json.load(open(p))['results']];print(sum(r['source']['kind'] == 'aggregator' for r in R), len(R), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results.
- **0.959 (vision2code-tools)** — The README's Top score for BenchCAD is 0.959 (vision2code-tools), from GPT-6 Astra (aggregator), the best row in its ledger. Metric: highest value in data/results/benchcad.json under the benchmark's metric, rendered as the README cell. Method: the receipt picks the top row again (value, ties to the earliest date), formats it with the benchmark's metric unit, appends conditions.split because this ledger carries rows on more than one split, and requires the README row for this benchmark to carry the same cell. Repro: python3 -c "import json, sys;B=json.load(open('data/benchmarks/benchcad.json'));R=json.load(open('data/results/benchcad.json'))['results'];T=min(R, key=lambda r: (-r['value'] if B['metric']['higher_is_better'] else r['value'], r['date']));V=('%g' % T['value']) + ('%' if B['metric']['unit'] == 'percent' else '');S={(r.get('conditions') or {}).get('split') for r in R} - {None};V+=(' (%s)' % T['conditions']['split']) if len(S) > 1 and (T.get('conditions') or {}).get('split') else '';C=[l for l in open('README.md', encoding='utf-8') if l.startswith('| [%s]' % B['name'])][0].split('|')[5].strip();print('%s: top %s · %s · %s · README cell %s' % (B['name'], V, T['system'], T['source']['kind'], C));sys.exit(0 if V == C else 1)". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/blob/main/README.md#model-benchmarks + https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results/benchcad.json.
- **72.6% (offline)** — The README's Top score for OSWorld 2.0 is 72.6% (offline), from GPT-6 Astra (self-reported), the best row in its ledger. Metric: highest value in data/results/osworld-2.json under the benchmark's metric, rendered as the README cell. Method: the receipt picks the top row again (value, ties to the earliest date), formats it with the benchmark's metric unit, appends conditions.split because this ledger carries rows on more than one split, and requires the README row for this benchmark to carry the same cell. Repro: python3 -c "import json, sys;B=json.load(open('data/benchmarks/osworld-2.json'));R=json.load(open('data/results/osworld-2.json'))['results'];T=min(R, key=lambda r: (-r['value'] if B['metric']['higher_is_better'] else r['value'], r['date']));V=('%g' % T['value']) + ('%' if B['metric']['unit'] == 'percent' else '');S={(r.get('conditions') or {}).get('split') for r in R} - {None};V+=(' (%s)' % T['conditions']['split']) if len(S) > 1 and (T.get('conditions') or {}).get('split') else '';C=[l for l in open('README.md', encoding='utf-8') if l.startswith('| [%s]' % B['name'])][0].split('|')[5].strip();print('%s: top %s · %s · %s · README cell %s' % (B['name'], V, T['system'], T['source']['kind'], C));sys.exit(0 if V == C else 1)". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/blob/main/README.md#agent-benchmarks + https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/results/osworld-2.json.
- **17/54** — 17 of the 54 benchmarks are recorded as saturated or retired, so they no longer separate frontier systems. Metric: benchmarks whose status is saturated or retired. Method: status field per benchmark file; active and saturating count as live. Repro: python3 -c "import json, glob;B=[json.load(open(p)) for p in glob.glob('data/benchmarks/*.json')];print(sum(b['status'] in ('saturated', 'retired') for b in B), len(B), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/benchmarks.
- **24/54** — 24 of the 54 benchmarks have a public, static, widely scraped test set. Metric: benchmarks whose contamination_risk is high. Method: contamination_risk field per benchmark file. Repro: python3 -c "import json, glob;B=[json.load(open(p)) for p in glob.glob('data/benchmarks/*.json')];print(sum(b.get('contamination_risk') == 'high' for b in B), len(B), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/benchmarks.
- **19/54** — 19 of the 54 benchmarks record a measured human baseline with a source. Metric: benchmarks with a non-null human_baseline (value, population, source). Method: human_baseline is only populated from a measured number with a citation; guesses are left null. Repro: python3 -c "import json, glob;B=[json.load(open(p)) for p in glob.glob('data/benchmarks/*.json')];print(sum(1 for b in B if b.get('human_baseline')), len(B), sep='/')". Evidence: https://github.com/alloevil/llm-benchmarks-tracker/tree/main/data/benchmarks.
- **0/252** — No receipt fetches a source page: all 252 rows state the number the page carried when it was read, and the gate only proves the committed value is the one the repository publishes. Metric: result rows whose source.url is fetched again while the gate runs. Method: the receipts are offline recounts of data/; re-reading the pages is scripts/sync_ledgers.py's job for the four allow-listed machine-readable sources and a person's job for the rest, and source.accessed records when either last did it. Repro: python scripts/sync_ledgers.py --dry-run  # needs the network; CI runs it twice a week. Evidence: https://github.com/alloevil/llm-benchmarks-tracker/blob/main/scripts/sync_ledgers.py.

## When to use it

- Deciding which benchmark actually measures a capability, and whether it still separates frontier systems.
- Checking where a quoted score came from: which kind of source published it, on what date, under which conditions.
- Reading benchmark metadata programmatically: 54 benchmarks and 18 evaluators as schema-validated JSON, free, no key, no rate limit.
- Following supersession chains (which benchmark replaced which) and human baselines (19 recorded with a citation).
- Citing a saturation or contamination judgement that is written down and reviewable rather than implied.

## When NOT to use it

- Not a model ranking. Rows within one benchmark differ in scaffold, reasoning effort, split and budget, so the top score is the best row in a ledger, not a statement that one model beats another.
- Not a live mirror of leaderboards. Structured sources are synced twice a week and everything else by hand, so a score published yesterday may not be here yet; read the accessed date on the row.
- Not a clean-room measurement. 55 rows are vendor self-reports and 13 come from aggregators republishing reported numbers; both are labelled, neither is independently verified here.
- Not a set of currently meaningful scores everywhere: 17 of 54 benchmarks are saturated or retired and 24 have a public, static, widely scraped test set, which is exactly why those two fields exist. Treat their scores as history.
- Not an evaluation harness. It cannot run a model; for that use one of the frameworks it catalogues.
- Not a cost or latency comparison. Cost appears only where a source stated it per task.

## Compared to Artificial Analysis, BenchLM, Epoch AI Benchmarking Hub, Scale SEAL Leaderboards and Vals AI

Those are evaluators and aggregators: they produce or republish scores. This project is a catalogue of the measurement landscape itself — one metadata file per benchmark plus provenance-labelled score ledgers — and it tracks each of them as an evaluator entry with its kind, maintainer, methodology and status. Use an evaluator when you need a fresh number produced under one controlled methodology; use this catalogue when you need to know which benchmark to look at, whether it is still discriminative, and where a number someone quoted at you came from. The evaluator entries record the distinction that matters when the same model shows two different scores in two places: whether the number was self-run, crowdsourced, submitted or merely collected.

## FAQ

**Does this project run the benchmarks?** No. It records numbers published elsewhere and labels each one with its source kind (official leaderboard, paper, independent evaluation, developer self-report, aggregator), the URL it came from and the date it was read there. If you need a number produced under a single controlled methodology, run one of the catalogued frameworks yourself or read an independent evaluator's own published data.

**How does a score get accepted?** Every row must name the system, the developer, the value, the publication date and a source with a URL, a kind and an access date; schema/results.schema.json makes all of that required, scripts/validate.py enforces cross-file invariants such as dangling benchmark references and impossible dates, and CI runs both on every pull request. Ledgers are append-only, so a corrected score is a new row with a newer date and the earlier row stays visible.

**What do the status values mean?** active means the benchmark still separates frontier systems; saturating means the top score is within roughly five points of the ceiling or of the human baseline; saturated means it no longer discriminates; retired means the maintainer stopped running it. For saturated and retired benchmarks the site shows the last reported score rather than a leaderboard top, because there is no meaningful current top.

**How do I get the data in bulk?** Every endpoint is listed at https://alloevil.github.io/llm-benchmarks-tracker/api/v1/index.json: benchmarks.json carries all benchmark metadata plus the top score and result count per benchmark, evaluators.json carries the evaluator catalogue, and results/<id>.json carries the full ledger for one benchmark. The JSON Schemas are published under /schema/, everything is static JSON on GitHub Pages, and the repository is MIT licensed, so cloning it instead is equally fine.

**How current is this file?** It is regenerated by scripts/build.py on every deploy from the same data/ directory as the site and the API, and was last built on 2026-09-16. The counts above are therefore exactly what the repository contained at build time, not a hand-maintained summary.