AI judges for LLM evaluation

An AI judge is a language model asked to grade another model's answers against a written rubric. It exists because the interesting properties of a good answer: correctness, completeness, tone, instruction-following, are not things a regex or exact match can check. When "correct" is a judgment call, you need something that can make judgment calls.

Two things decide whether judge scores mean anything: the rubric (what the judge is told to look for) and the judge model (whether it can actually make the call). Most bad eval results trace back to one of the two, not to the candidates being tested.

This guide is the deep dive on the grader. If you are still deciding whether to run a custom eval at all, start with how to evaluate an LLM on your own data, which covers the full workflow: cases, trials and reading the numbers.

Every row produces one judged score per candidate per trial. Because the judge sees the same rubric every time, differences between candidates come from the candidates, not from an inconsistent grader. That is the point, and it only holds if the rubric is written well enough to apply consistently.

A weak rubric is "Rate the answer 1-5". A judge can and will make up what those numbers mean, differently on every row. A strong rubric names the criteria and anchors each score level with a concrete description, so the judge is matching the answer to descriptions instead of inventing a scale.

A rubric that works has four parts:

1. What a perfect answer does, stated positively: "includes all required fields, no extra commentary, cites the input row it used."
2. Anchored score levels, not a bare scale. Describe what a 1, 3 and 5 look like for this task specifically.
3. Automatic failures: the things that make the score irrelevant, like leaking a required format or contradicting the input.
4. The output contract: tell the judge to return only a score and a one-line reason, or it will ramble and you cannot parse it.

Score this answer 1-5 on instruction-following and completeness.

5: answers every part of the request, correct format, no invented details
3: answers most parts OR correct content with format violations
1: misses the request, contradicts the input, or invents facts
0: refuses, errors, or empty

Task: {task}
Answer: {answer}

Return ONLY JSON: {"score": n, "reason": "one line"}

The {task} and {answer} placeholders get filled per row. That template above is roughly the shape of what orbench sends when you enable AI judging; the important part is that it lives in your benchmark file, so it is versioned and frozen with the rest of the eval.

Judges have systematic, documented biases. They are not random noise, which means they push results in predictable directions, usually toward whichever candidate writes longer, sounds more confident, or was generated by the judge's own family.

A judge cannot grade what it cannot do. Asking a small, cheap model to grade expert-level reasoning produces confident wrong scores: it cannot tell a subtly broken answer from a correct one. The rule of thumb from the eval guide: the judge should be at least as capable as the strongest model it grades.

If the honest answer is "no available model can grade this task", that is a real finding: the task needs human review or deterministic checks, not a bigger judge.

Before scaling to hundreds of rows, score a sample by hand and compare:

1. Pick 15-30 cases from the set, across easy and hard ones.
2. Score them yourself (or with a teammate) before looking at judge scores.
3. Compare: agreement rate and, more important, where it disagrees. Random scatter means the rubric is vague; a cluster on one failure mode means the judge cannot see that failure mode.
4. Fix the rubric or the judge, re-run the sample, repeat until agreement is acceptable for your decision.

You do not need 95% agreement. You need the judge to rank candidates the same way a human would, because the decision downstream is comparative: "model A beats model B on our tasks". A judge that is consistently harsh but consistently ordered is usable; one that flips rankings is not.

Fully human review does not scale; fully automated judging hides judge errors. The working split: the judge grades everything, humans arbitrate the close calls and audit a sample.

That last row is how orbench's human review works: candidate labels are hidden, you score without knowing which model wrote what, and the verdicts slot into the report next to the judge's. If the two disagree systematically, that is a signal about the judge, not about the candidates.

• Code can verify it. Exact answers, required fields, parseable output: deterministic checks are cheaper, faster and cannot be biased.
• The task is at the edge of the judge's own ability. Expert legal reasoning, novel proofs, obscure domain facts: the judge will miss the same errors the candidates make.
• The decision does not need it. If a model fails 40% of rows on a format check, you already have your answer without judging a single row.
• The judge call costs more than the signal. Judge tokens run once per row per candidate. On a tight budget, a smaller sample judged well beats the full set judged poorly.

What is LLM-as-a-judge?

Using a language model to grade another model's answers against a written rubric, instead of checking outputs by hand. It is the only grading method that scales to open-ended answers where correctness is not a string match.

Can the same model be candidate and judge?

Avoid it. Models rate their own outputs generously (self-preference bias), so a candidate judging itself inflates its score. Pick a judge stronger than the candidates, ideally from a different family.

How do I stop judge biases from skewing results?

Blind the inputs, randomize order in pairwise comparisons, write anchored score levels instead of a bare 1-5 scale, and spot-check a sample of judge decisions against human review.

When should I not use an AI judge?

When code can verify the answer, when the task is at the edge of the judge's own ability, or when the judge call costs more per row than the value of the extra signal.

How do I know my judge can be trusted?

Score a sample by hand, compare judge verdicts on the same cases, and look at where disagreements cluster before scaling up.