# agent-harness-evals

agent-harness-evals is a reconciliation layer over public AI evaluation leaderboards that exposes the model x agent-harness matrix model-only leaderboards hide: which harness gets the most out of each model, and how much the harness alone moves the score.

## What it is

Not a benchmark. An aggregation and reconciliation layer over benchmarks that already exist. Public leaderboards are single-layer: they rank models, and the agent harness (scaffold) that produced each score is either fixed or invisible. This project keeps the harness as an axis. It fetches published leaderboard data, normalizes model and harness identifiers so the same model from two sources joins into one row instead of fragmenting, and renders per-benchmark views no single leaderboard offers: the model x harness matrix, a harness ranking (the share of each model's best score a harness retains), a model ranking (each model's best score and the harness that got it there), and native-pairing boards where every model runs only in its vendor's own CLI. As of 2026-10-24 the corpus holds 7005 normalized records over 94 benchmarks; 10 of those benchmarks have a genuine cross-harness matrix (41 models, 50 harnesses), and 3 are native-pairing boards (FrontierCode, weirdml_v3, PostTrainBench).

### Where the numbers come from

Every score here was published by someone else; this project measures nothing and runs no models. Sources are the Epoch AI Benchmarking Hub (a CC-BY data dump refreshed about daily, which carries the agent x model boards including Terminal-Bench, BTF3, FrontierCode and PostTrainBench) and the HAL leaderboards from Princeton (public HTML score tables across nine benchmarks, a historical harness snapshot rather than a live feed). Upstream publishes benchmark-level aggregates, not per-trial results, so every cell is the best published score for that (model, harness) pair and `spread` is computed from those aggregates, with the number of harnesses a model was measured under carried alongside as `n`.

### How it refreshes

.github/workflows/refresh.yml runs daily at 05:23 UTC: fetch, normalize, rebuild the harness matrices, write a dated JSONL snapshot, then regenerate docs/index.html, docs/trend.html and the retrieval artifacts (llms.txt, llms-full.txt, claims.json) and commit only if something changed. The page is therefore exactly as fresh as upstream, at zero evaluation cost, and the numbers in this file, in claims.json and on the page all come from the same build payload, so they cannot disagree.

## Install

```bash
git clone https://github.com/alloevil/agent-harness-evals
cd agent-harness-evals
pip install -r requirements.txt
```

Nothing needs installing to read the results: the site inlines its whole dataset, and the matrices are committed as CSV under https://github.com/alloevil/agent-harness-evals/tree/master/views.

## Quickstart

```bash
python src/fetch.py           # download Epoch dump + HAL pages into data/raw/
python src/normalize.py       # -> data/clean/records.parquet, one unified record schema
python src/views_harness.py   # -> views/harness_matrix_<benchmark>.{csv,md}
python src/snapshot.py        # -> data/snapshots/<date>.jsonl
python src/build_site.py      # -> docs/index.html, llms.txt, llms-full.txt, claims.json
python src/build_trend.py     # -> docs/trend.html
```

## Verifiable claims

Machine-readable copy: https://alloevil.github.io/agent-harness-evals/claims.json. Every value is read off the same payload the page was built from, so it matches what the site shows. All of it is aggregation of third-party leaderboard scores, not measurement performed here.

- **7005** — 7005 leaderboard records across 94 benchmarks are normalized into one (benchmark, model, harness, score) schema. Metric: rows in the daily snapshot data/snapshots/2026-10-24.jsonl, before per-view (model, harness) dedup. Method: third-party leaderboard scores fetched from the Epoch AI Benchmarking Hub and the HAL leaderboards and normalized into one record schema; no evaluation is run by this project; model ids are canonicalized so the same model from two sources joins instead of forking. Repro: pip install -r requirements.txt && python src/fetch.py && python src/normalize.py && python src/views_harness.py. Evidence: https://github.com/alloevil/agent-harness-evals/tree/master/data/snapshots.
- **41 models x 50 harnesses** — 41 models and 50 harnesses appear in at least one cross-harness matrix, over 10 benchmarks. Metric: distinct models and harnesses on boards where a model was measured under 2 or more harnesses. Method: aggregation of published leaderboard scores (Epoch AI Benchmarking Hub, HAL): best published score per (model, harness) pair; no evaluation is run by this project; models measured under a single harness are excluded from the matrix. Repro: pip install -r requirements.txt && python src/fetch.py && python src/normalize.py && python src/views_harness.py. Evidence: https://github.com/alloevil/agent-harness-evals/tree/master/views.
- **animated and reducible** — Changing the sort reorders the table with FLIP (WAAPI) and the page honours prefers-reduced-motion — the animation only moves the same numbers. Metric: docs/index.html contains a motion primitive and a reduced-motion branch. Method: build_site.py animates row positions and bar widths with element.animate() after a re-render; the printed values are rendered, never tweened, and a reduced-motion media query disables it. no evaluation is run by this project. Repro: src/build_site.py (FLIP helpers). Evidence: https://github.com/alloevil/agent-harness-evals/blob/master/docs/index.html.
- **0.134** — On Terminal-Bench, the median gap between a model's best and worst agent harness is 0.134 (score as a 0-1 rate), across 22 models measured under 3 or more harnesses. Metric: median over models with 3+ harnesses of (best harness score - worst harness score) for that model. Method: aggregation of published leaderboard scores (Epoch AI Benchmarking Hub, HAL): best published score per (model, harness) pair; no evaluation is run by this project; models with fewer than 3 harnesses are excluded because a 2-harness gap is a single pairwise difference. Repro: pip install -r requirements.txt && python src/fetch.py && python src/normalize.py && python src/views_harness.py. Evidence: https://github.com/alloevil/agent-harness-evals/blob/master/views/harness_matrix_terminalbench.md.
- **0.289** — On CORE-Bench Hard, the median gap between a model's best and worst agent harness is 0.289 (score as a 0-1 rate), across 3 models measured under 3 or more harnesses. Metric: median over models with 3+ harnesses of (best harness score - worst harness score) for that model. Method: aggregation of published leaderboard scores (Epoch AI Benchmarking Hub, HAL): best published score per (model, harness) pair; no evaluation is run by this project; models with fewer than 3 harnesses are excluded because a 2-harness gap is a single pairwise difference. Repro: pip install -r requirements.txt && python src/fetch.py && python src/normalize.py && python src/views_harness.py. Evidence: https://github.com/alloevil/agent-harness-evals/blob/master/views/harness_matrix_hal_corebench_hard.md.
- **0.031** — On SciCode, the median gap between a model's best and worst agent harness is 0.031 (score as a 0-1 rate), across 7 models measured under 3 or more harnesses. Metric: median over models with 3+ harnesses of (best harness score - worst harness score) for that model. Method: aggregation of published leaderboard scores (Epoch AI Benchmarking Hub, HAL): best published score per (model, harness) pair; no evaluation is run by this project; models with fewer than 3 harnesses are excluded because a 2-harness gap is a single pairwise difference. Repro: pip install -r requirements.txt && python src/fetch.py && python src/normalize.py && python src/views_harness.py. Evidence: https://github.com/alloevil/agent-harness-evals/blob/master/views/harness_matrix_hal_scicode.md.

## When to use it

- Choosing an agent harness for a model you have already picked, or a model for a harness you are locked into.
- Sanity-checking a leaderboard number before quoting it: the same model may sit points away under another harness on the same benchmark.
- Budgeting how much of a score difference is attributable to scaffolding rather than to the model.
- Getting a reconciled, machine-readable corpus of (benchmark, model, harness, score) rows without scraping several leaderboards yourself.

## When NOT to use it

- Not a measurement. Nothing here is re-run; if an upstream leaderboard is wrong, this site is wrong in the same way, and each row keeps its source so you can check.
- Not trial-level data. Upstream publishes aggregates, so spreads are differences between published scores, not paired per-task comparisons, and confidence intervals exist only where a source reported them.
- Not equally solid everywhere. Only 3 of the 10 cross-harness boards have any model measured under 3 or more harnesses; on the rest (GAIA, SWE-bench Verified Mini, τ-bench Airline, Online-Mind2Web, ScienceAgentBench, BTF3, USACO) a spread would be a single pairwise difference, so no median spread is published for them at all.
- Not a live harness feed beyond one source. Terminal-Bench via Epoch is the only live, machine-readable harness leaderboard; the HAL dimension is an imported historical snapshot, and boards that pin a single harness (SWE-bench's official board runs mini-swe-agent for everyone) cannot produce a harness matrix at all.
- No tool-layer comparison (tool A versus tool B under a fixed agent): no public data source for it exists yet.
- No contamination or saturation labels. This project does not judge whether a benchmark is still discriminative; it reports what harnesses did on it.

## Compared to the Epoch AI Benchmarking Hub, HAL and the Messier corpus

The Epoch AI Benchmarking Hub is the upstream data source here and is live, but it is organised as boards per benchmark: the harness axis exists only where a board happens to be an agent x model board. HAL published harness x model x cost tables across nine benchmarks and is the richer harness source, but it is a snapshot rather than a live feed. The Messier corpus is a task- and verifier-level harness corpus and is the most detailed of the three, but its dataset has no public download yet, so it cannot be imported. This project is the combination none of them offers on its own: cross-layer, in one reconciled schema, and refreshed daily. It adds no scores of its own — for original numbers, read those sources directly.

## FAQ

**Does this project run benchmarks or agents?** No. It downloads scores that leaderboards already published, reconciles the model and harness names behind them, and pivots them into matrices. There is no evaluation cost, no API key and no model access anywhere in the pipeline, which is also why every limitation of the upstream boards is inherited here.

**What exactly is a harness?** The scaffold around a model: the agent loop, tool set, prompts, retry policy and step budget — Claude Code, Codex CLI, Droid, OpenHands, mini-SWE-agent, Terminus and so on. In the record schema it is the `scaffold` field, and an empty scaffold means the provider default (an unscaffolded model score), which is kept distinct from a named harness.

**Why do some benchmarks show no median spread?** Because no model on that board was measured under three or more harnesses. With two harnesses the difference is one pairwise comparison, which a median cannot make more reliable, so the number is withheld rather than presented as a spread. The matrix itself is still shown, with each model's `n` visible.

**How current are the numbers?** The pipeline runs daily at 05:23 UTC and commits only when something changed; this file, claims.json and the page were all generated from the payload dated 2026-10-24. Historical values stay available as one JSONL snapshot per day under data/snapshots, which is what the trend page reads.

**Can I reuse the data?** The code and the derived views are MIT licensed and the daily snapshots are committed in the repository, so cloning is the simplest route. The underlying scores belong to their publishers: Epoch AI's dump is CC-BY and asks for credit, and HAL's tables belong to the HAL maintainers, so keep those attributions when you republish.