How to read public benchmarks

A public benchmark measures a model on a fixed set of tasks under a fixed scoring rule. That is genuinely useful for two things:

• Shortlisting. If a benchmark's tasks resemble your workload, its leaders are worth a closer look.
• Tracking capability. Watching which benchmarks are still hard tells you where the frontier is and where it has already saturated.

It cannot tell you whether a model will do your job. Your prompts, your formats, your tolerance for failure and your budget are all absent from the leaderboard.

Each entry in the benchmarks directory answers the questions that decide whether a score means anything:

• What it tests. The description says what the tasks actually are. "Coding" covers everything from autocomplete to multi-day repo work: read what the task looks like, not just the category.
• Scale. The task count. Twenty hand-checked tasks can outsignal a thousand scraped ones; check both size and provenance.
• Who maintains it. A lab's benchmark for its own model deserves more skepticism than an independent one. Continuous benchmarks that patch and re-release tasks stay meaningful longer.
• How it is scored. Exact match, rubric judge, hidden verifier or human vote each favor different model styles. A pairwise arena and a pass/fail task suite are not interchangeable numbers.
• Public or private tasks. Private or regularly refreshed task sets resist contamination. A public set that has been on GitHub for a year may be in the training data.
• The headline score. Look at the ceiling, not just the leader. If the best model scores 35%, the benchmark still discriminates. At 95%, it mostly measures noise.

None of these mean a benchmark is useless; they mean read the score carefully before you let it decide anything:

• Contamination. Tasks and answers that leaked into training data inflate scores. Newer and private-set benchmarks suffer less.
• Setup differences. The same benchmark reported with different agents, tool budgets or prompt setups is not the same number. Check which setup produced the score before comparing two models.
• Version drift. SWE-bench, Terminal-Bench and others ship new versions constantly. A score from v1 is not comparable to v4.
• Averages hiding failure modes. A model can top an average while failing every case of the type you care about. Per-category and per-task results beat the headline.
• Cost left out. The cheapest way to win many benchmarks is more reasoning tokens. Two models at the same score can differ 10x in cost per task.
• Benchmaxxing. Labs optimize for reported benchmarks. Benchmarks that mutate tasks or audit reward hacking exist precisely because published scores can be gamed.

A practical workflow:

1. Filter to your category. Pick the section closest to your workload: coding, agents, domain work or general reasoning.
2. Check recent releases first. The newest benchmarks are the least saturated and least contaminated. The ticker at the top of the directory tracks them.
3. Pick two or three relevant benchmarks, not one. A model that leads across several task types you care about is a stronger signal than a single leaderboard win.
4. Take the top candidates plus one wild card. Include a cheaper model: the job of your own eval is to find out if the extra cost buys real quality.
5. Test the shortlist on your own tasks. This is the step public benchmarks cannot do for you. Twenty to fifty of your real cases, one rubric, one run.

Should I trust leaderboards from the model labs themselves?

It depends less on who built the leaderboard than on how transparently it was run. A lab's own benchmark is not automatically suspect; an opaque one is, whoever made it. Before trusting any leaderboard, check what you can actually inspect:

• Are the tasks published? If you cannot read what the model was asked, you cannot judge whether the test resembles your workload. Private sets are fine for contamination reasons, but only when paired with sample tasks or a public subset.
• Is the scoring method documented? Look for the rubric, the verifier, or the judge prompt. "We ran an eval" without a visible grading process is a claim, not a result.
• How was the test conducted? Which agent harness, tool budget, prompt and effort setting. Two scores on the same benchmark under different setups are not comparable, and the flattering setup is the one that gets reported.
• Can third parties reproduce it? Open datasets, open scoring code and published trajectories let others check the numbers. Independent reproduction is the strongest trust signal a leaderboard can offer.

In practice: benchmarks built by the lab that tops them are worth extra scrutiny, not automatic dismissal. Several widely used benchmarks started that way. The line to draw is between evaluations you can inspect and ones you have to take on faith.

How do I know if a benchmark is contaminated?

You usually cannot prove it. Signals that help: public task sets over a year old, sudden jumps when a new model ships, and models that do well on the public split but poorly on the private one. Fresh and private-set benchmarks are the safer read.

Is the number one model on the leaderboard the one I should pick?

Not on leaderboard rank alone. Rank plus task relevance, cost and the failure modes you can tolerate picks the shortlist; your own eval picks the winner.

Why does the directory track release dates?

Because freshness is a quality signal. A new benchmark still discriminates between frontier models and is less likely to be in training data. An old benchmark at 95% tells you little beyond "everyone passes".