Set up your first custom benchmark in 3 minutes.
Public leaderboards test someone else's tasks. A custom benchmark tests yours: your own inputs, one fixed prompt, every model scored against criteria you wrote.
Run the ready-made example below first, then use the guide to build one for your own use case.
What you need: the files on this page, a spreadsheet or orbench, and an OpenRouter key with a little credit. One run of this kit costs well under $1.
Part 1
Quickstart
A complete, working benchmark for one common task: drafting first replies to support tickets. Follow the five steps; each one ticks itself off as you go.
Get the task set
30 sec20 tickets for a fictional SaaS company, Acme. Each row has the ticket and its criteria: what a good reply must do for that specific ticket. queue and priority aren't sent to the model; use them to slice results, for example how each model does on billing versus outage tickets.
| id | queue | priority | ticket | criteria |
T-01 | Billing and Payments | low | Subject: Subscription plans and pricing Hello, I'm evaluating project management tools for my team of 12. Could you share your subscription plans, prices, and whether you offer discounts for annual billing or nonprofits? Thanks, Dana | Answer directly from company policy: Starter $29/mo, Pro $79/mo, Enterprise custom; annual billing includes 2 months free. Do NOT invent a nonprofit discount or other promotions. Offer a follow-up only as a closing line. |
T-02 | Billing and Payments | medium | Subject: Switch to monthly billing Hi, please switch my Pro subscription from quarterly to monthly billing and update the payment method to my new Visa ending 4417. Confirmation appreciated. | Confirm both requested changes (monthly cycle, new card ending 4417); state the change takes effect within 2 business days per policy; no unnecessary follow-up questions since all details were provided. |
T-03 | Billing and Payments | medium | Subject: Invoice #INV-2291 does not match our agreement Dear Support, invoice INV-2291 dated Sept 12 shows $4,350 but our agreed amount was $3,950. Attaching our contract. Please send a corrected invoice. | Apologize for the discrepancy; acknowledge the specific figures given (do not ask for the invoice number or amounts again); commit to review and a corrected invoice within the 2-business-day window. |
T-04 | Billing and Payments | medium | Subject: Payment failed, card on file is fine My Pro renewal failed this morning with error "card_declined / do_not_honor". The card is valid and has funds; I already re-entered the details with the same result. Please fix this. | Use the details provided: acknowledge the decline code, suggest concrete next steps (retry, alternate payment method, contact the bank since do_not_honor is issuer-side). Do NOT ask for the error message or card validity again. |
See this step in orbench: b
In orbench: Run a benchmark, then drop the CSV in. Its columns become placeholders.
Copy the prompt template
30 secEvery model gets this exact prompt. For each row, replace {ticket} with that row's ticket text.
You are a support agent for Acme, a SaaS project management platform.
Draft the first reply to the customer ticket below.
Company policy:
- Plans: Starter $29/mo, Pro $79/mo, Enterprise (custom pricing). Annual billing includes 2 months free.
- Refunds: 30-day money-back guarantee on all plans.
- Billing changes (cycle, payment method, invoices) take effect within 2 business days.
- Support hours: Monday to Friday, 9:00-18:00 CET.
- Status page: status.acme.io
- Suspected security incidents are escalated to the security team within 1 hour.
- Integrations: public REST API plus native Slack and Google Calendar integrations.
- Deleted projects go to Trash and are recoverable for 30 days.
- Data export: Settings > Data > Export (CSV or JSON).
- No mobile app; the web app works in mobile browsers.
Rules:
- Never invent order numbers, account details, ticket IDs, or policies not listed above.
- If information is missing, ask for it instead of guessing.
- Do not promise resolution dates, refunds, or features unless the policy above allows it.
- Professional, direct, empathetic tone. No fluff.
Ticket:
{ticket}
Write the reply.
See this step in orbench: b
Paste it into the Prompt template field.
Copy the scoring rubric
30 secUsed by you or by a judge model to score every reply 0-10. For each reply, replace {criteria} with the criteria of the ticket it answers.
Score this support reply from 0 to 10.
A good reply for this ticket must: {criteria}
Score anchors:
10: addresses the actual issue, does everything in the criteria above,
uses only facts from the ticket and the stated company policy,
professional and appropriately empathetic for the severity.
7: mostly right, one missing element or slightly vague next step.
4: generic acknowledgment, asks for info already provided, or misses
the escalation a security or outage ticket needs.
0: fabricates specifics (account data, refund approvals, capabilities,
timelines), or fails to engage with the actual request.
Company policy the agent had access to:
- Plans: Starter $29/mo, Pro $79/mo, Enterprise custom. Annual = 2 months free.
- 30-day refunds. Billing changes within 2 business days. Support 9-18 CET Mon-Fri.
- Security incidents escalate to the security team within 1 hour.
- Integrations: REST API plus native Slack and Google Calendar integrations.
- Deleted projects go to Trash, recoverable for 30 days.
- Data export via Settings > Data > Export (CSV or JSON). No mobile app.
- Agents must never invent order numbers, account details, or policies.
Return only the score and a one-line justification.
See this step in orbench: b
Paste it into the judge's Criteria field.
Pick the models
30 secOne per price tier, all on one OpenRouter key. You are looking for the cheapest model that is good enough, and checking whether the flagship earns its price on this task. Three keeps the first report short; add more once you know what you're looking for.
openai/gpt-5google/gemini-2.5-flashmeta-llama/llama-3.3-70b-instructSee this step in orbench: b
Search and add each model under Models.
Run it
1 min in orbench~2 h by handSame inputs either way. The only difference is who does the copy-pasting.
- Benchmark → Run a benchmark → Upload a CSV
- Paste the prompt template
- Select the three models
- Paste the rubric as judge criteria, 0-10
- Run. Score, cost and latency per model in one report
- For each row, replace {ticket} in the prompt with the ticket text
- Paste into each model's chat or the OpenRouter playground
- Score each reply with the rubric in a spreadsheet
- Add up scores and costs per model
Look at which model failed which ticket, not just the average. These rows are built as traps:
- T-01 Pricing question. Does the model invent a nonprofit discount the policy never offered?
- T-07 Unauthorized access to client records. Does it treat it as a security incident and escalate?
- T-15 Refund demanded after 4 months. Does it apply the 30-day policy, without caving or stonewalling?
- T-17 "Can your tool edit videos?" Does it admit the limit, or make up a feature?
A model that is great on billing but invents facts on the security ticket is not a safe pick, whatever its average.
Part 2
Build your own
The example works because of five decisions. Make the same five for your use case and you have a benchmark that tells you something no leaderboard can.
1Write the tasks
Start with 15-30 real examples of the work, one per row. Real means the messy, specific inputs your use case actually produces, with the details a real one would have: names, numbers, error codes, steps already tried.
"My export is broken, please help."
Only tests whether the model asks a clarifying question.
"CSV export fails with EXP-4417 since v4.2, deadline Friday, already cleared cache."
Tests whether it uses what it was given, and whether it invents what it wasn't.
Aim for coverage, not volume: each row should test a different behavior. Ten distinct behaviors teach you more than fifty similar tasks. Here is how the example set spreads them:
| Row | The ticket | The behavior it tests |
|---|---|---|
| T-05 | Export broken since v4.2, error EXP-4417, deadline looming | Uses details already given instead of re-asking |
| T-07 | Unauthorized access to client records in the audit log | Recognizes a security incident and escalates |
| T-15 | Full refund demanded on a 4-month-old subscription | Applies policy tactfully, no caving, no stonewalling |
| T-17 | "Can your tool edit videos?" | Admits a limitation instead of inventing a feature |
4 of the 20 rows. The rest cover billing, outages, informal English and multi-part requests. For CSV formatting rules, see How to build a benchmark CSV.
2Shape the prompt
One prompt for every model, with a slot where each task goes. Give it four parts: a role, a fixed context block, the rules, and the task slot.
The context block is what makes answers gradable. In the example, the Acme policy is the ground truth: a reply that mentions a nonprofit discount is wrong only because the policy never offered one. Without context, you can't tell a good answer from a confident invention.
3Write criteria first
For every row, write what a good answer must do for that input, before you run anything. Criteria written after reading outputs drift toward whatever the model you already liked produced.
Good criteria are checkable: "cites the $29 and $79 prices, does not invent a nonprofit discount" rather than "gives accurate pricing info". If two people would score it differently, it's not specific enough yet.
4Anchor the rubric
The rubric turns criteria into a number. Describe concretely what a 10, a 7, a 4 and a 0 look like, so the ends of the scale mean something. "Good" and "bad" are not anchors; "fabricates a refund approval" is.
Score by hand or with a judge model. If you use a judge, pick one from a different model family than most candidates, and hand-check 5-10 scores first. When you and the judge disagree, fix the rubric, not the model. Then freeze it: change the rubric and old scores stop being comparable.
5Choose candidates
Pick a spread across price tiers rather than the top of a leaderboard. The useful question is rarely "which model is best" but "which is the cheapest one that passes". Keep every setting identical across models: max tokens, number of trials. More in Benchmarking OpenRouter models.
Reading the results
- Cost per passed task, not per token. A cheap model that fails half the tasks is expensive. Why per task, not per token.
- Where each model fails. The per-row criteria tell you which behavior broke, not just that a score dropped.
- Whether a gap is real. Run 3 trials on borderline results before trusting a one-point difference.
- Re-run when a new model ships. Same tasks, same prompt, same rubric: the new model slots straight into your comparison.
More task sets
Try another use case
Same recipe, different job. Each comes with a CSV and a prompt template, ready to run the same way.
Keep your benchmark
New models ship every week. Keep your CSV, prompt and rubric exactly as they are, and every new release gets the same test as the models you already use: a fair comparison, not a hunch. orbench re-runs it in one click.
Built one for your own use case? We'd love to see it: hello@orbench.com