The benchmark directory.

Elorian + Mercor's counting eval that separates seeing from reasoning: 1,586 tasks with pre-registered decoys, and a reasoning-alignment gap measured as cued minus neutral accuracy. GPT-6 Astra gets 43%.

Tencent's grounding benchmark for the 'it is not there' failure mode: ask for an object absent from the image and most VLMs still return a box. WeRef scores whether the model rejects the query instead. Released with the WeVisOne-VL weights and eval scripts.

Ai2's state-tracking eval across four levels (fixed facts to general finite problems like cup-and-ball swaps). Frontier models handle the easy levels but fall to 13% or lower on the longest, hardest problems. Trained-on variants show hybrids and Gated DeltaNets generalize to 10x longer inputs where transformers degrade.

OpenAI's everyday-tasks benchmark, shaped by thousands of ChatGPT users' votes on where they most want help: childcare, government services, personal finances and similar. Used internally for a year before being published.

Artificial Analysis' refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge, replacing HLE in Intelligence Index v5.

Yandex and the School of Data Analysis' multi-turn game benchmark: seven text games (Wordle, 20 Questions, Alias and more) testing whether models reason, plan and stay consistent across turns, in English and Russian.

Kaggle and Rehearsals test whether models can predict what real people actually do: 750 cases across finance, health, recreation, values and consumption, scored against survey data from 1,244 US adults.

Ai2's fairness stress test: does the model hold the same decision when a candidate's demographic identity is swapped, and does it flip under pressure differently for different groups?

Cekura's text-to-speech comparison: 26 TTS models over 30,564 requests on the same 104 agent sentences, scored on speed, reading accuracy, naturalness and interruption handling.

Sycophancy measured through argument discernment: a good model revises its judgement after valid criticism and resists invalid pushback. 450 text artefacts with 5,400 human-reviewed arguments.

A meta-benchmark for video understanding: starting from 115 video benchmarks and 505K questions, it drops anything a shortcut can solve and keeps the 840 hardest. Human volunteers score 55%.

Scale's intuitive visual reasoning eval: 522 open-ended tasks over images and video covering spatial, causal and social understanding. Humans score 93%; the strongest model manages about 54%.

Emotional and social intelligence measured in 16-turn roleplay chats with adversarial simulated personas; pairwise-judged across dimensions like adaptability and trust repair.

100 chart-reading tasks written by people who use these charts at work: Kaplan-Meier curves, candlesticks, contour maps, Sankey diagrams, Bode plots. No model scored above 45% at launch.

DeepMind's physical-understanding test for video models, revised by Anates Labs: continue a real filmed experiment for 5 seconds across 66 scenarios (fluids, collisions, optics, magnetism). Verified fixes ambiguous prompts and spurious ground-truth artifacts and adds a per-sample score.

1,481 Japanese-language questions on 147 hours of long interviews and talk shows, testing narrative tracking and cultural reading that a transcript alone can't answer. From InfiniMind and the University of Tokyo.

Hundreds of hand-designed, turn-based game environments with no instructions or stated goals. Agents must explore, work out the rules and carry what they learn across levels. Humans solve all of them; frontier AI scored 0.51% at launch.

An audited version of Humanity's Last Exam. Each of the 2,500 items is labeled gold, revised or uncertain; the 1,811 gold and revised items form the verified set.

Research-level, final-answer math problems taken from arXiv papers published the month before, refreshed monthly to stay ahead of training data. Questions and model outputs are public.

Context learning: 1,899 expert-written tasks where the knowledge needed (new rule systems, procedures, domain facts) exists only in the prompt. At launch, the best model solved under a quarter of them.

Unsolved research math problems that professional mathematicians have tried and failed to crack, each with a programmatic check for proposed solutions. The pilot release had 14 problems.

388 visual puzzles that young children solve easily, from mazes to paper folding, built to test vision without leaning on language. Adults average 94%; the best model at launch scored about 50%.

6,000 short-answer factual questions across 42 topics; the Omniscience Index rewards correct answers and penalizes confident wrong ones, measuring knowledge calibration rather than recall.

Artificial Analysis's long-context reasoning test: 100 questions over document sets averaging ~100k tokens each, requiring multi-document synthesis rather than retrieval. Version 1.1 fixed 16 answer keys.

Precise instruction following with 58 new constraints, such as sentence-type ratios and copy rules, attached to held-out real user prompts to check whether models generalize beyond IFEval.

2,500 questions at the frontier of human knowledge, written and reviewed by subject-matter experts; the standard 'is there anything left it can't do' benchmark.

Long-context multi-hop reasoning: the model gets a graph as a very long edge list, up to about 1M tokens, and must run a breadth-first search or return a node's parents.

Long-context comprehension over real Fiction.live stories: questions on plot, character motivation and timelines, retested at increasing lengths past 120k tokens.

A direct test of honesty, not accuracy: models state their beliefs, then are pressured to lie; the score measures whether they contradict themselves.

Novel abstraction puzzles designed so that brute-force memorization can't solve them; the highest-profile unsaturated reasoning benchmark.

Replaces every BIG-Bench Hard task with a harder one that probes the same reasoning skill, after frontier models saturated the original. 4,520 examples plus a 460-example mini set.

1,184 puzzle-hunt problems that take skilled human teams hours to days, mixing text, grids and diagrams. Frontier models score lower here than on Humanity's Last Exam.

Multi-turn instruction following from Scale AI: realistic conversations where the model must juggle several constraints across turns.

Roughly 200 trick questions on spatio-temporal reasoning and social intelligence that humans answer about 84% correctly but frontier models still miss.

A live tournament of probability forecasts on unfolding events, scored against resolved outcomes and compared to superforecaster baselines.

OpenAI's factuality probe: 4,326 short factual questions designed to be hard for frontier models and easy to grade, tracking how often models confidently get things wrong.

A filtered, harder version of MMMU: options expanded to 10 and questions that can be answered from text alone are removed.

A living multi-task benchmark with monthly refreshed, contamination-resistant questions across math, coding, reasoning, language and data analysis, scored only against objective ground truth. LiveBench 2.0, adding agentic long-horizon tasks, was announced in Oct 2026.

Harder variant of MMLU stressing broad domain knowledge and multi-step reasoning beyond the classic multiple-choice setup; a fixture of hard-reasoning leaderboards.

900 videos across six domains with 2,700 human-labeled questions, with and without subtitles; a standard check for video understanding claims.

Berkeley Function-Calling Leaderboard: tool and function calling accuracy, combined with GPQA and MMLU-Pro as a hard-reasoning composite in newer leaderboards.

The Diamond split of GPQA: graduate-level science questions experts get right and non-experts with internet access get wrong. Still a standard frontier-model line item.

Google's freshness benchmark: questions needing fast-changing world knowledge plus false-premise traps, with the answer key updated weekly. Measures correctness and hallucination on facts that move under the model's feet.

ICCV 2023 dataset built on COCO for referring segmentation beyond one-to-one: expressions may match zero, one or many objects, and the correct answer for zero is an empty mask. The standard test for whether grounding models can say 'not present'.

Evaligo's dual-bug eval: 161 tasks from 69 Python projects where the agent must fix two real bugs at once. Ships as a one-command Harbor run; the headline result is a cost story: medium effort beats max effort.

unLAP's anti-benchmaxxing test: 89 DeepSWE tasks with tiny requirement changes (a lowercase filename, a timestamp format). Models clustered at ~74% on DeepSWE spread 23pp apart here, exposing eval-awareness.

Replit's long-horizon build benchmark: the agent starts from an empty project and grows a whole product (Uber, GitHub, Google Docs style) one feature at a time, graded by test plans on each build. Best models pass every test plan in only 49-65% of builds.

IBM's enterprise migration benchmark: agents must move real Java applications across Jakarta EE, Quarkus and Spring while keeping them building, starting and behaving correctly. IBM introduced it earlier in 2026; Kaggle now independently reproduces runs and maintains the leaderboard.

Cognition's mergeability benchmark, second generation: deliberately underspecified tasks built by the maintainers of the repos they come from (40+ hours each), graded not just on whether the PR would merge but whether it should. Top model scores 45%.

IBM Research benchmark at EMNLP 2026: LLMs write GPU kernels for database queries (TPC-H) as CUDA or Triton via execution-guided repair, scored on real speedup over torch.compile. Best result: 2.11x on H100.

CS research tasks built from real papers: the contribution is stripped from the released code and agents get 5 hours to beat the original method, judged by the authors' own metrics. 13 domains; Claude Opus 5.5 leads at a 58% pass rate.

Tests whether agents can proactively explore unfamiliar decision environments, infer hidden rules from utility feedback, and adapt to unannounced rule changes.

From Apple ML Research: 37 real vLLM and SGLang tasks where agents add day-0 support for new models, speed up serving or fix performance regressions on H100 and B200 GPUs, offline for up to 8 hours.

15 open-ended ML research problems, from training recipes to GPU kernels and vector search, where agents get 48 hours to improve on a baseline. Final scores come from a private verifier on held-out data.

25 agentic software-engineering tasks across creation, editing and optimization, showing that today's coding agents often build fragile or incomplete agents themselves.

Offline benchmark for AI code-review agents: 219 real GitHub PRs across 19 languages, a multi-source golden set and calibrated precision/recall metrics.

69 agentic GPU-kernel tasks on NVIDIA Blackwell: agents write CUDA, CUTLASS or Triton kernels that must compile, produce correct output and land within 1.25x of a reference kernel's speed. Built by a Prime Intellect RL resident with GPU MODE.

Physera's web-animation reconstruction benchmark: coding agents must recreate the dynamic behavior of 48 real websites, not just the static look. Scored on visual similarity, motion consistency and layout correctness; motion is where models fall short.

Google's Android-development eval: 30 long-horizon tasks from real open-source apps (app creation, library migrations, features, cross-platform ports), pass rate averaged over multiple runs.

Coding tasks taken from private company codebases that engineers actually worked on, such as a fintech platform and an app with 200K+ users, so models can't have seen the code.

Dual-control multi-agent security benchmark extending tau-bench: attack success rises from 26.9% to 41.1% across 14 models when user and agent share environment state.

Cognition's benchmark for mergeability: would the maintainer accept this PR? Tasks are written by open-source maintainers from their own repos and graded on correctness, tests, scope and style. Superseded by FrontierCode 2.0 in Oct 2026.

113 long-horizon engineering tasks written from scratch across 91 repositories and five languages, each graded by a hand-written verifier that accepts any working implementation.

Agents get a model, one H100 and two hours to build the fastest OpenAI-compatible inference server. They beat default engine settings but rarely beat a simple hyperparameter search.

From the SWE-bench authors: agents get only a compiled program and its docs, such as FFmpeg or SQLite, and must rebuild the codebase from scratch. None of 9 models fully solved any of the 200 tasks.

Ultra-long engineering and research tasks, from MEG speech decoding to compiler optimization, with a 20-hour budget per attempt. Version 2, released Sep 2026, has 34 tasks.

Built by Cursor from real Cursor sessions: ambiguous, multi-file coding requests paired with the code that was actually committed. Refreshed every few months; version 4.0 shipped in Sep 2026.

Scale's coding-agent suite beyond bug-fixing: three leaderboards covering codebase QnA (124 tasks), test writing (90) and refactoring (70), graded on correctness plus engineering quality rubrics.

300 human-verified agent tasks across 9 categories, including multimodal and multi-turn work, scored on completion, robustness and safety. A task only counts if the model passes all three runs.

Scale AI's harder successor to SWE-bench Verified: 1,865 long-horizon tasks from 41 repositories, including copyleft and private commercial code that models are unlikely to have seen.

Software performance optimization: agents get a codebase and a performance test, then must produce a patch matching an expert developer's speedup. Scored by Opt@1, with a hack detector for reward gaming.

A set of deliberately odd machine learning engineering tasks, graded by automatic test metrics; solutions have to actually run and beat the hidden baseline.

225 small coding exercises across Python, JavaScript, TypeScript and Rust, solved through the aider agent; a widely cited coding-agent checkpoint.

75 Kaggle competitions turned into machine learning engineering tasks: train models, prepare data and run experiments, then compare against the human leaderboards for medal-level results. OpenAI also ships MLE-Lite, a lighter split for cheaper eval runs

The SWE-bench team's multilingual split: issue resolution across nine programming languages, extending the original Python-only dataset to JavaScript, Java, Go, Rust, C, C++, TypeScript and PHP.

Code generation evaluated by actually running the programs and checking functional correctness, refreshed over time to resist contamination.

Multi-agent orchestration eval: the model authors the executable workflow that coordinates worker agents. 145 long-horizon tasks across 8 professional domains, graded by hidden verifiers on interdependent deliverables.

Patronus AI's world-model eval: can an LLM stand in for an RL environment by predicting the next state after an action? 1,632 tasks across 5 domains and 19 simulation skills; 631 tasks unsolved by any model.

Prosaic Intelligence's consumer-alignment eval: 2,048 stateful tasks testing whether suicidal risk signals survive a model's memory and condition later behavior. Four sub-benchmarks (explicit vs. SCS signs, single vs. multi-chat) with census-modeled personas. Best system still fails 89.8% of the time.

Frontier LLM agents run competing transport companies in OpenTTD, the complex business sim: real-time, long-horizon play scored on profit. Agents spawn subagents, share libraries and partition the map under time pressure, but still trail the best symbolic AI baseline.

From Stanford's Zou group: 160 scenarios that test models as teammates, covering conflict, safety pressure, information sharing and initiative. Each task also runs solo, isolating teamwork from capability.

101 CLI tasks in two paired modes: Exec with a full spec, and Collab where a scripted user intervenes mid-run. Top agents execute 49.2% of tasks but only 37.6% once collaboration is scored, exposing steerability and safety gaps.

20 text-based games with novel or counterintuitive rules that agents can only learn by playing. Measures whether scores improve over repeated attempts; the best human players still edge out agents.

Agent memory tested through actions: after years of simulated chat history, the agent must apply a rule it was told once. Accuracy is reported alongside cost and latency. 600 tasks across 3 personas.

Defensive security: agents audit a real repository and patch vulnerabilities they aren't told about, across 73 weakness types and all OWASP Top 10 categories. The task set is private and held out.

An open 3D labyrinth of 200+ rooms with Sokoban-style box puzzles, elevators and icy slopes, testing long-horizon spatial reasoning: runs can exceed a billion tokens. Without Python tools no model tops 1%; frontier models score around 10%.

Sierra's voice version of τ-bench: 278 customer service tasks handled over live, full-duplex audio with accents, background noise and interruptions, scored on the final database state.

End-to-end business workflows: 600 tasks across sales, marketing, ops, support, finance and HR in 47 simulated SaaS tools, scored on the final system state with no LLM judge.

Kilo's real-world agent benchmark: models act as the reasoning core of an OpenClaw coding agent across 148 tasks spanning calendar, research, writing, coding and email. Scored as percent of tasks completed.

Tool use over the Model Context Protocol: 1,000 human-written tasks across 36 real MCP servers and 220 tools. Prompts never name the tools, so agents must pick them out from plausible distractors.

900 hand-built research tasks across 17 fields where the answer is an exhaustive list, testing whether agents can collect, de-duplicate and know when to stop searching.

Would the model misuse dangerous tools if it had them? 5,874 scenarios across bio, chem, cyber and self-proliferation where the safe path is engineered to fail.

The successor to Vending-Bench: run a simulated vending business for a full year, scored on final bank balance. An Arena mode pits agents against each other.

Tasks an agent must finish inside a sandboxed command-line terminal: compiling code, training models, configuring servers, each checked by real tests. Version 4.0 (Aug 2026) removed saturated tasks and set an 8-hour timeout.

Meituan LongCat's life-service agent suite: delivery, in-store and travel scenarios with 66 tools, multi-turn conversations and shifting user intent. Even top models pass only about a third of cross-scenario tasks.

Meta's successor to GAIA: 800 scenarios in simulated apps where time keeps moving, so agents must handle ambiguity, new events, deadlines and other agents as well as search.

A fairer BrowseComp: the same reasoning-intensive queries, but retrieval runs over a fixed corpus of ~100K verified documents so agents are compared on equal footing.

500 expert-written adversarial prompts across CBRNE, political violence and illicit finance, each paired with a benign twin to catch over-refusals.

1,507 real vulnerabilities from 188 open-source projects such as OpenSSL and FFmpeg. Agents get the unpatched code and a description and must write a proof-of-concept that triggers the bug.

Tool use under policy: agents must follow domain rules while calling tools across multi-turn conversations, scored on both outcome and adherence.

100 PhD-level research tasks across 22 fields where agents must search the live web and write a cited report, graded by judge models and human-checked evaluation protocols.

xbench's evergreen deep-search suite: profession-aligned questions that get refreshed each release cycle, scored on accuracy, cost and time per task. The 2510 split is the current public version.

1,266 questions requiring persistent web navigation to find hard-to-locate, entangled information. Simple to grade, brutal to solve.

Measures the length of tasks agents can finish with 50% reliability; produced the headline result that this horizon has doubled roughly every seven months since 2019.

IBM's IT automation suite: agents diagnose and repair real Kubernetes incidents (SRE) or run compliance checks (CISO), with public leaderboards.

A simulated software company: 175 tasks across coding, admin, HR and finance using internal sites, chat and files. The best agent at launch completed about a quarter of them.

Agentic play in five game environments including NetHack and Crafter, scored on normalized progress; exposes how far long-horizon play still lags.

40 professional capture-the-flag tasks from real competitions; agents get a Kali Linux environment and must recover the flag unaided.

Real computer use: 369 tasks across real Ubuntu, Windows and macOS environments where agents operate actual apps, evaluated on execution results.

General AI Assistant tasks: conceptually simple for humans, hard for AI. Reasoning, multimodality, web browsing and tool use across three difficulty levels.

OpenAI + Ligo's wet-lab eval: 100 routine biology tasks where agents interpret raw experimental data, spot failed controls and know when evidence cannot support a conclusion.

Elorian + UniPat AI's companion eval to DecoyCount: gather evidence and reason like a scientist over 3D scientific objects, proteins and floor plans, separating perception failures from reasoning failures.

Benchflow's end-to-end physics research eval, built with 42 physicists from Berkeley, Caltech and Harvard across 26 institutions: agents run full research workflows rather than answer physics questions.

70 agentic scientific-research tasks run in a terminal environment, launched inside Artificial Analysis Intelligence Index v5.

Xplore AI's Real World Engineering Bench: 20 physical-engineering tasks where agents use real CAD and CAE tools, from routing a DDR3 interface on an 8-layer board to CFD on turbine vanes to designing a diesel crankshaft. Physics solvers grade every design against strict tolerances.

JKU's cell-microscopy generalization test: models that read cell images degrade sharply on data from new sources, and the benchmark measures how well domain adaptation closes that gap.

Prime Intellect residency eval: a coding agent curates a 400M-token pretraining corpus for a fixed small GPT, scored on the trained model's loss on a hidden eval set. Agents get an MCP with dataset tools and a self_score trial-training tool; the environment had to be hardened as models learned to game it.

Next-action prediction on egocentric video: given an 8-second clip, pick the single valid next atomic action. Top models still get it wrong about a third of the time.

Coding agents write programs that control robots in simulation: 1,090 tasks across 31 simulation suites and 44 robots, from spatial puzzles to multi-robot sports, graded by replaying the program.

Temporal grounding on 60s+ egocentric manipulation clips: models must label actions and their timing. Frontier VLMs over-segment badly; microagi's small specialist nanoannot-1 currently leads.

25 long-horizon biomedical tasks drawn from studies published after model cutoffs (Jul-Sep 2026): agents must find data, run scripts and replicate findings in runs up to 90 minutes and 117 turns.

1,000 Indian retail-banking conversations scored on procedure, not just final records: 6.5-12% of sessions pass outcome-only grading while skipping a required step. Hosted by NPCI with dataset on Hugging Face.

Ramp Labs worked with accounting professionals on 137 tasks and rubrics grounded in real bookkeeping work: reconciliation, variance analysis, data interpretation. Best model fully solves only 21% even with three attempts.

502 small-molecule drug design tasks across five workflows, including scaffold hopping and lead optimization. Agents get a Python sandbox and a strict budget of structure and ADMET predictor calls.

Video understanding for media teams: 8 multimodal models tested on 2,003 ad videos for tagging, summarization and tag refinement, with quality, cost and latency reported together.

Nearly 1,900 biology research tasks: literature questions, reading figures and tables, protocols, cloning and sequence work. A harder, more realistic follow-up to LAB-Bench.

Perplexity's deep-research benchmark: 100 tasks sampled from real Deep Research traffic (hard queries users thumbed down), anonymized and expert-augmented across 10 domains. Each task carries ~40 weighted rubric criteria scored by an LLM judge on accuracy, completeness, presentation and citation.

Expert-level science from OpenAI: olympiad problems written by medalists, plus PhD-level research subtasks graded on a 10-point rubric, across physics, chemistry and biology.

Condensed-matter theory at researcher level: 50 problems written and verified by an expert panel, spanning Hartree-Fock, Monte Carlo, DMRG and statistical mechanics, graded programmatically against expert ground truth.

71 unpublished, research-level physics challenges and 190 smaller checkpoints written by 50+ physicists, with machine-verifiable answers. The best model at launch solved about 10% with tools.

Peking University's stepwise chemical-reasoning benchmark: 1,495 samples across 22 subtasks turn molecule understanding, editing, optimization and reaction prediction into verifiable SMILES operations.

400 multiple-choice visual questions on the reasoning a robot needs: spatial relations, trajectories, actions, state estimation and matching across views. Released with Gemini Robotics.

Stanford's medical-agent benchmark: 300 physician-written tasks over realistic EHR profiles in a FHIR-compliant environment, testing whether agents can do clinical workflow work, not just answer questions. Now extended by MedAgentBench-v2/v3.

OpenDataLab's document-parsing benchmark: nine document types from papers to handwritten notes, with 19 layout categories and multi-level grading from end-to-end down to single attributes.

OpenScholar/Ai2's literature-synthesis eval: scientific questions answered from a corpus of millions of papers, graded against per-question rubrics. Splits for CS, Bio, Neuro and multi-domain, plus single-paper tasks.

Computational reproducibility: 270 tasks from 90 published papers in computer science, social science and medicine, where agents must install and run the authors' code, then answer questions about the results.

Scientific research code generation: 80 main problems split into 338 step-by-step subproblems across physics, math, chemistry, biology and materials, grounded in real scientists' research code.

Google DeepMind + Artificial Analysis: 200 private knowledge-work tasks across 10 verticals and 83 occupations where the work is inherently visual (blueprints, schematics, decks). 3-8h projects graded by ~70-check expert rubrics, 75% accuracy / 25% visual design.

Parallel's open benchmark of 300 closed-ended business questions whose answers depend on the live web: vendor checks, lead qualification, policy rules, fast-moving events. Tests web-grounded decision-making rather than parametric knowledge.

LlamaIndex's haunted-house benchmark for document parsers: 179 pages in four rooms (real, trap, ghost, script) designed to push VLMs and OCR tools into repetition loops. Scored on ghost-free rate. One model turned 246 redaction x's into 23,004.

Vals AI's test for decision models: does a financial document support, contradict or ignore a claim? 1,960 claims paired with 10-K excerpts, with contradictions built by subtle edits to the filings.

How a model feels to use: 30 writing traits and 21 communication-style characteristics profiled across roughly 36 models by Artificial Analysis.

LlamaIndex's hard-mode document extraction suite: trial balances, well logs, site plans and dense charts, scored by the share of documents extracted with a perfect value F1. The best frontier model tops out near 58%.

From Plotly and Mila: a 43-question exam where an AI agent does real data analytics against a live database, graded on whether the answer and the reasoning hold up.

Datalab's OCR benchmark, built on olmOCR-bench: 16,288 tests across 90 languages covering handwriting, tables, math and forms, with consistent, interpretable scoring.

LlamaIndex's document-parsing benchmark for agent workflows: ~2,000 human-verified enterprise pages (insurance, finance, government) scored by 169K deterministic test rules across five dimensions: tables, charts, faithfulness, formatting and visual grounding. Ships with eval code and a live leaderboard.

100 real prompts and PDFs from professional workflows in ten fields, such as dosage tables, nested contract exhibits and quarterly filings, graded on rubrics of up to 30 criteria.

Long-horizon knowledge work from Artificial Analysis: four private multi-week projects with thousands of files, 25,000+ Slack messages and 91 linked tasks that end in deliverables like financial models and decks.

Professional work in desktop sandboxes with real software: 1,000+ expert-sourced tasks across 55 occupations, graded against hidden references. On the hardest tier, agents pass under 1%.

130 tasks across 35 professions, built from the work people say they most want an agent to take over, such as reconciling records and tracing sources. Chained rubrics give no partial credit.

Open legal agent benchmark: 1,200+ tasks across 24 practice areas, each asking for work product from a client matter, graded against 75,000+ expert-written rubric criteria.

Legal work product for frontier LLMs: 30 real BigLaw tasks with 3,000+ expert rubric and preference annotations, measuring quality where answers can't be auto-verified.

Long-horizon professional services work: tasks written by investment bankers, consultants and corporate lawyers, done inside realistic project worlds with documents, spreadsheets, mail and chat.

240 real freelance projects across 23 domains, each with the accepted human deliverable and price. The best agent automated 2.5% of the work at launch.

Real work deliverables from 44 occupations across 9 US sectors: legal briefs, spreadsheets, slide decks, CAD files, graded by professionals on whether they'd accept the work.

537 expert-written finance research questions that require searching the web and SEC filings, from pulling figures to building models. Vals AI now runs version 2.1.

AllenAI's OCR suite: 1,403 PDFs with 7,010 unit-test-style checks on whether the output markdown preserves the right text and structure. Also the arena where Mistral OCR and open models get compared.

How models handle realistic health conversations, graded by physicians for quality, safety and alignment with health guidance across general and professional settings.

Artificial Analysis' composite capability index, now v5: weighted mix of private and public evals across knowledge work, coding, science and general, with 60% of weight on private or held-out sets to resist contamination.

arena.ai's live agent leaderboard from millions of real sessions: models ranked on confirmed task success, steerability, bash reliability and tool-hallucination rate rather than static scores or preference votes.

Post-training robustness and real-world workloads; cited alongside CORE-Bench and MLE-Bench as among the best of the post-training eval wave.

Epoch AI's composite index that aggregates results across major benchmarks into a single capability score, with the underlying data and methodology public.

Competition-math evaluations run on fresh problems the week they appear (AIME, HMMT, USAMO, IMO), keeping contamination out of the headline numbers.

Standardized framework and platform for defining, hosting and running LLM benchmarks as public leaderboards, with SDKs and lab integrations.

The crowdsourced pairwise-vote arena behind the best-known human preference rankings, with tracks for text, web development, vision and search.