How to evaluate an LLM on your own data

Strip away the jargon and an LLM evaluation is three things:

1. Cases. A fixed set of inputs: the support tickets, documents, questions or prompts your system actually handles.
2. A pass rule. For each case, a definition of a good answer. Sometimes that is an expected answer to match against; sometimes it is a rubric describing what "good" looks like.
3. A grader. Something that applies the rule identically to every answer: code for anything code can check, a judge model for the rest, a human where it matters most.

That is it. Running the eval produces a score per model and, more usefully, a list of exactly which cases passed and failed. The score lets you compare; the list tells you why.

Eval vs benchmark vs leaderboard

The three words get used interchangeably and they are not the same thing:

A leaderboard is a benchmark rendered as a ranking; an eval is a benchmark you own. The rest of this guide is about the third row.

Not every model decision needs a custom eval. A public benchmark is enough when the capability it measures is the thing you care about: if you are picking a model for generic reasoning or trivia, the standard test suites have thousands of cases and years of validation behind them.

You need your own eval when any of these are true:

• Your task has vocabulary, formats or rules no public benchmark covers: house style, a specific taxonomy, a required output shape.
• Public benchmarks are saturated for your capability: every frontier model scores 95%+ and the ranking tells you nothing about your workload.
• Your real question is not "which model is best" but "which model does this well, cheaply and fast enough".
• You are changing prompts or providers and need to know whether anything got worse, not whether it got better on average.

The step most teams skip is also the one that decides whether the eval is worth anything: writing down what a good answer looks like before looking at any model output. If you write the pass rule after seeing the answers, you will unconsciously grade toward whichever answer you liked.

The exercise takes half an hour and needs no tooling:

1. Write down 5 real inputs your system has handled.
2. For each one, write the output you would accept and the output you would reject.
3. List the failure modes you care about: invented facts, wrong format, too long, off tone, refuses when it should answer.

If you cannot do this in 30 minutes, your success definition is too vague to measure, and no amount of eval tooling fixes that. If you can, you have just written the seed of your test set and your rubric at the same time.

The cases matter more than the tooling. Mix four kinds:

Two rules of thumb on sourcing. First, real traffic beats invented cases: an export of tasks your system actually handled, with the outcomes you know were right, tests what is hard in production rather than what you imagine is hard. Second, load in the failures deliberately: a set where every model scores 95% costs the same to run as a useful one and tells you nothing.

Twenty to fifty cases is enough to compare models and catch regressions. Fifty real rows beat five hundred synthetic ones.

Match the grader to the kind of quality you are checking:

Most evals end up hybrid: code rejects the structural failures, a judge scores the rest, a human spot-checks the judge. A judge's reliability comes down to two decisions: a rubric anchored in concrete descriptions rather than adjectives, and a judge model at least as capable as the strongest model it grades. Judges also have known biases worth designing around; they favor longer answers, first-seen answers, and sometimes their own family's writing style.

The mechanics of writing a rubric a judge can follow, with worked examples, are in the benchmarking guide.

Run every candidate model against the same set, same prompts, same template, and more than once per case: models answer the same prompt differently call to call, so a single trial lets one lucky generation skew the ranking.

Then read three numbers together, not one: the rubric score, the answer rate (how often the model returned a usable answer at all), and cost per answered task (total spend divided by tasks that got an answer). The best-scoring model is not always the right pick: a model a few points behind on quality but half the cost and twice as fast is often the better production choice.

Execution detail lives in the other guides: the full workflow, why cost per answered task is the number that matters, and a real report showing what the output looks like.

An eval's value compounds: the same set, re-run when a new model ships or a prompt changes, tells you whether anything actually improved. That only works if the measuring stick stays fixed.

• Freeze the cases, the prompt template, the judge model and the rubric. Changing any of them changes what the scores mean.
• If you need to add tasks, version the set (tickets-v2.csv) and treat it as a new baseline rather than quietly editing the old numbers.
• Keep the eval in version control next to the code it tests. It is a test suite, not a spreadsheet in someone's Downloads folder.

When to re-run

An eval earns its keep on the second run, not the first. Re-run when:

• A new model ships that could replace your current one
• You change the prompt, the system prompt, or the provider routing
• Your provider silently updates the model behind the same endpoint
• Production traffic shifts: new case types arriving means the old set may no longer represent the workload

Saved runs make re-running cheap: in orbench a past benchmark can be replayed against a new model with the same rows, judge and criteria, then compared side by side with the original. The workflow details are in benchmarking OpenRouter models on your own tasks.

The failure modes are mostly not technical:

• Teaching to the test. If the eval cases leak into the prompt ("always do X because case 14 checks X"), scores rise while production does not. The eval should measure the system, not steer it.
• A set that is too easy. Every model at 95% is a measurement of nothing. The discrimination lives in the past-failure and edge rows; prune cases that stop differentiating, or weight them down deliberately.
• Stale cases. An eval built from last year's traffic measures last year's workload. When production shifts, refresh a slice of the set rather than letting it fossilize.
• Grader drift. Swap the judge model or rewrite the rubric and every historical score becomes incomparable. Freeze the grader with the cases, or accept that you are starting a new baseline.
• Trusting the judge without checking. Read a sample of judged answers on every new eval. Judges confidently grade wrong on tasks they misunderstand, and you will not notice from the aggregate score.
• Optimizing the score, losing the point. The score is a proxy for "will this model do the job". When the proxy and the job diverge, fix the eval, not the model choice.

What is an LLM evaluation?

A repeatable test of a model on a fixed set of cases, where each case has an input and a rule for what counts as a good answer. It gives you a score per model and a list of exactly which cases passed and failed.

When do I need a custom eval instead of a public benchmark?

When your task has vocabulary, formats or quality rules public benchmarks do not cover, or when your real question is whether a model does your specific job well, cheaply and fast enough. If a public benchmark already measures what you care about, use it.

How many test cases do I need?

Twenty to fifty well-chosen cases are enough to compare models and catch regressions. Mix typical inputs, edge cases, past failures and a few adversarial ones: the discrimination comes from the hard rows.

Should I grade with code or a judge model?

Code for anything it can verify (exact answers, formats, required fields), a judge model with a written rubric for semantic quality. Anchor the rubric's scale with concrete descriptions and pick a judge at least as capable as the strongest candidate.

How do I keep results comparable over time?

Freeze the cases, template, judge and rubric. Version the set rather than editing it, and keep it in version control like any other test suite.