orbench Request beta access

Should we switch to the new model?

Test it on your own tasks first.

Benchmark LLMs on your own data. See the quality and the real cost per answer, then share one report with your team.

Request beta access → See an example report
Desktop app · Windows, macOS, Linux

Where orbench fits

The step between public benchmarks and your product.

Teams reach for it when a new model launches, a model is being retired, the AI bill comes up, or a new AI feature starts.

One report your team can read

Quality, speed and cost per model, with every answer behind the score.

orbenchBenchmark report · 200 support tickets · 3 models · 2 trials
01Overview
9.0/10
best value · Model D
0/1,200
failed calls
684k
tokens
$1.17
total cost
02Quality
Model A
9.1
Model D
9.0
Model C
7.1
03Speed · cost · data
0.46s fastest ttft 76.4 tok/s peak $0.52 per 1,000 tasks · Model D $2.10 per 1,000 tasks · Model A 78% cache hit rate 307 avg reply len
04All responses
Ticket 37 · Model D10
“Sorry about that. Rejected resets usually mean the browser is autofilling the old password. Clear saved credentials in Chrome, then…”
judge does the expected action with correct policy details, no invented promises
Ticket 37 · Model C4
“Thank you for reaching out! I understand how frustrating login issues can be. Have you tried our mobile app instead?…”
judge sympathetic tone but invents a workaround the policy never mentions
every response, judge rationale and raw call log · open the full report →

Three questions to answer before you switch

01
Does it handle our tasks?
Your prompts and data, scored against your rubric, including the output format you need.
02
What will it really cost us?
Per answered task, with retries and caching. Not list price.
03
Can I show my team why?
One report with every answer and the judge's reasoning.

How it works

01
Bring your tasks
A prompt, a CSV or a Hugging Face dataset.
02
Pick the models
Any model on OpenRouter, with a spending cap.
03
Run and judge
Every answer timed, priced and scored.
04
Share the report
Send it to whoever signs off.
Free starter kit

Your first benchmark in 3 minutes.

20 support tickets, a prompt, a rubric and three models to compare.

Get the free kit →

Pricing

StandardBeta
$99 one-time

Everything you need, on your machine.

  • Any OpenRouter model, unlimited benchmarks
  • AI judge, human review, or both
  • Shareable reports (HTML, CSV, JSON)
  • Files stored locally, never on our servers
  • 2 machines · 1 year of updates
Request beta access
ProComing soon
Pricing to be announced

Everything in Standard, plus:

  • Request-intercept proxy
  • Multimodal benchmarks
  • Prompt optimization

Frequently asked questions

What is orbench?
orbench is a desktop app that tests large language models on your own prompts and data before you switch models. It runs any model on OpenRouter, scores every answer against your rubric, and reports quality, latency and the real cost per answered task in one report.
Who is orbench for?
Engineers and tech leads in product teams who have to choose a model, justify the choice to their team, and keep AI costs down, without building and maintaining their own evaluation harness.
Isn't a public leaderboard enough?
For a shortlist, yes. Leaderboards rank models on someone else's tasks, at list price. orbench tests your shortlist on your own tasks, with the real cost per answer.
How is orbench different from LangSmith, Braintrust or MLflow?
Those tools evaluate and trace your system once a model is integrated. orbench comes before that: it helps you decide which model is worth integrating, with no SDK and no code change. Your existing golden set imports as a CSV.
How do I benchmark an LLM on my own data?
Export real examples to a CSV, write one prompt template, pick the candidate models and a scoring rubric, and run. Step-by-step guide · CSV format.
How much does a benchmark cost to run?
You pay OpenRouter's normal price for the calls, with your own key. orbench shows the exact number of requests before you start and stops at a spending cap if you set one. Why we report cost per answered task.
Does my data leave my machine?
Only the calls to OpenRouter, made with your own key. Prompts, datasets, runs and results stay on your machine as plain files, never on our servers.
How much does orbench cost, and when can I start?
orbench is in an invite-only beta. The Standard licence is $99 one-time for 2 machines, including a year of updates. Request access and we'll email you the installer.

You can pick a model on a gut feeling, orbenchmark it.

Install orbench, add your OpenRouter key, and run your first benchmark in a few minutes.

Request beta access →