What 167 benchmarks tell us about AI evaluation in 2026

The directory tracks 167 public benchmarks at the time of writing, and the age profile is striking: 96 of them, nearly 60%, were released in 2026 alone. Only five entries predate 2024.

By modality the picture is even clearer: 74 entries evaluate agents rather than raw model outputs. The field has largely stopped asking "what does the model answer" and started asking "what does it get done".

Every benchmark in the directory, whatever its shape, is built from the same four parts. Knowing them makes every card in the index easier to read:

1. The task. What the model is asked to do. This moved from single questions (MMLU, SimpleQA) to multi-step jobs that take a professional hours: build a working feature, run a real accounting workflow, conduct a physics investigation.
2. The environment. What the model can act on. Early evals were text in, text out. Now 74 entries give agents terminals, browsers, FHIR servers, CAD tools, simulated SaaS stacks or live web access.
3. The scorer. How an attempt is judged. Exact match and test suites came first; then rubric-graded judges; now hidden verifiers, physics solvers, pairwise votes and auditors that inspect the whole trajectory for reward hacking.
4. The score. What the leaderboard actually reports: pass rate, rubric average, Elo, wall-clock cost, token spend, or composite indices like the AA Intelligence Index that bundle many evals into one number.

When two benchmarks disagree about a model, the disagreement almost always lives in one of these four parts: different tasks, different environments, different scorers or different metrics. That is also why the same model can rank first on one leaderboard and tenth on another.

Searching the directory's evaluation notes shows how fragmented scoring has become. A single method rarely dominates a benchmark, so these counts overlap:

Two things stand out. First, the judge is now part of the score: roughly a quarter of the directory grades with another model, which means a leaderboard position partly measures who wrote the rubric. Second, human grading did not go away, it moved upmarket into the hardest evals, where a wrong answer looks right to anything but an expert.

Not the frontier labs, mostly. Of the entries that name a maintainer, only about 20 list a frontier-model lab: OpenAI appears on ten (often with partners), Google and DeepMind on around ten between them, Anthropic on one. Universities account for another ten or so.

The other ~130 come from the eval ecosystem itself: specialist eval shops (Scale AI and Artificial Analysis each appear on eight), startups measuring their own domain, and platforms turning production traffic into leaderboards. Benchmarking has become its own industry, sitting between the labs and the buyers.

That split matters when you read a score. A lab's benchmark is self-measurement and deserves the scrutiny we covered in the reading guide. An eval shop's benchmark is a product, which means it is built to be useful to buyers but also to be legible to them. Neither motive is disqualifying, but both shape what gets measured.

The second thing the data shows is how short a benchmark's useful life has become. Terminal-Bench went through four major versions in a year. SWE-bench Verified, arguably the most-cited coding eval of 2025, is retired. Two entries in the directory are already marked superseded.

Three forces kill benchmarks faster than they used to:

• Contamination. A public task set that has been on GitHub for a year is probably in somebody's training data. The tell is unexplained jumps on old benchmarks while harder fresh sets stay flat.
• Benchmaxxing. Labs tune for what gets reported. When DeepSWE's mutator benchmark MaxxBench changed small details in 89 tasks, models clustered at 74% on the original spread 23 points apart, which is a direct measure of eval-specific training.
• Saturation. Once the leader scores 95%, the remaining signal is noise. Benchmarks that stay useful either keep getting harder or measure something that resists saturation, like cost or failure rate.

The interesting pattern in the recent releases is not the scores, it is the design. Six shifts stand out:

Grading the process, not just the answer

Newer evals check how a model works, not only whether it finished. TauIndianBankBench grades the procedure an agent follows. Physics solvers verify CAD output in RWE-Bench. MaxxBench asks whether a model follows changed instructions rather than replaying a memorized solution. An answer can be right for the wrong reason, and these evals are built to catch that.

Private and rolling task sets

Twenty-nine entries now keep their tasks hidden, private or continuously refreshed to resist leakage. LiveBench regenerates monthly. Google's MKA keeps all 200 tasks private. The trade-off is real: private sets stay clean but cannot be inspected. The healthier pattern is a private set plus public samples or a public subset.

Continuous benchmarks

Terminal-Bench, BFCL and the SWE-bench family now ship like software: versioned releases, patched tasks, hardened verifiers. Terminal-Bench 4.1 exists almost entirely to make reward hacking harder. A benchmark with a changelog ages far better than a frozen dataset.

Live evals and arenas

Twenty entries are arenas or continuously updated leaderboards. Agent Arena scores models on 2.7 million real production sessions rather than fixed tasks: measured steerability, tool reliability and confirmed success in the wild. It is a different kind of evidence, less controlled but harder to game.

Behavioral failure modes

A new family of evals measures what models do wrong, not what they can do: WeRef tests whether a model will say "this object is not in the image" instead of hallucinating a box. Suicide Memory Bench tests whether risk signals survive in an assistant's memory. RealCLI found models drop roughly 12 points when a task requires collaborating with a user rather than executing alone. Capability benchmarks tell you the ceiling; these tell you the floor.

Evals that fight reward hacking

The newest category is the eval whose purpose is to catch eval-gaming itself: task mutation (MaxxBench), reward-hacking audits that inspect trajectories rather than outputs, and hidden verifiers designed to survive attempts to game them. The community's word for the problem is "benchmaxxing": training a model to the benchmark rather than to the skill. That dedicated evals now exist just to detect it is itself a statement about how much published numbers can be trusted.

The short version, by example

Three practical takeaways:

1. Recency is a quality filter. A model leading a benchmark released this quarter is a stronger signal than one topping a saturated 2024 set. The directory's ticker exists for exactly this reason.
2. Match the eval to your workload, not the hype. If your product does document work, a 200-task multimodal knowledge benchmark outranks ten coding leaderboards for your purposes.
3. Public scores pick the shortlist, never the model. Every design trend above, private sets, procedure grading, arenas, exists because published numbers diverge from real-world behavior. The last step has to be your own tasks.

Everything above comes from the 167 entries in our public benchmark directory, each card sourced from the benchmark's own paper, repository, dataset or leaderboard. A few honest limits:

• Public only. This counts what organizations chose to publish. Evals that labs run privately, or that customers pay to keep private, are invisible here.
• Recency skew. The directory is newest-first by design, and the field's acceleration (96 entries in 2026) means our picture of "the past" is thin. Older evals we lack may not be unimportant, just unindexed.
• Counts are keyword-based. The scoring-method table searches evaluation notes for terms like rubric, judge, verifier and arena. Benchmarks using several methods count under each, and loosely worded notes can miss a method entirely. Treat those numbers as directional, not exact.
• Private-set claims are taken at face value. When a benchmark says its test set is private, we record the claim; there is no way for a third party to verify what a lab does not publish.

How often does the directory get new entries?

In a heavy release week we add five to ten. The past two months saw roughly twenty new benchmarks, most of them agentic or domain-specific.

Are older benchmarks worthless?

No, but read them differently. A mature benchmark with a public set and years of results is useful for tracking capability over time, less useful for choosing between frontier models today.

Why do so many benchmarks come from labs themselves?

Because labs need evals to ship, and publishing them buys credibility. It is not automatically suspect, but check transparency: published tasks or samples, documented scoring, reproducible results.

Where does orbench fit?

The directory tells you what the field measures and who leads it. orbench is the next step: running your shortlist on your own tasks, with your rubric, before you commit.