Model evaluation, hosted
Find the best model for your agentic work.
Measured on your own tasks, in real sandboxes. Amber runs Model Configurations on real Tasks and shows what each one achieves, and what it costs.
No card needed. See the full pricing.
How it works
From “which model?” to an answer you can defend.
- 1
Pick what to measure
Choose public Benchmarks such as Terminal-Bench and SWE-bench, or bring your own Tasks as a private Benchmark.
- 2
See the Quote first
Amber estimates the cost of the Job before anything is spent, and that Quote becomes its spending ceiling.
- 3
Run Trials in real Environments
Each Trial is one Agent and Model Configuration attempting one Task in an isolated sandbox, judged by the Task’s Verifier.
- 4
Compare like for like
Rewards are compared only when Benchmark Version and Agent Version match, with confidence intervals and a “too close to call” flag.
$ amber quote -b terminal-bench@2.0 -m opus-high -m gpt-high
Quote $184.20 (based on 140 past Trials)
$ amber run -b terminal-bench@2.0 -m opus-high -m gpt-high
Accept this Quote? [y/N] y
Job 7f3a started · 178 Trials · streaming progress…Illustrative output. The same flow is available in the web app and the API.
Why Amber
Built for the choice, not the leaderboard.
- Real work, real Environments
- Agentic Tasks run in sandboxes and a Verifier inspects the end state. A Model either got the job done or it did not.
- Your own Benchmarks, kept private
- Upload Harbor Task folders or draft Tasks with guided authoring. Private Benchmarks are visible only to your Workspace.
- Pinned, so results hold up
- A Benchmark Version is an immutable snapshot and an Agent Version is pinned, so a Reward from March still means the same in October.
- Honest statistics
- Mean Reward comes with a 95% bootstrap confidence interval. When two Model Configurations cannot be told apart, Amber says so.
- Cost is a first-class axis
- Model, sandbox and judge costs are itemised per Trial, so the question is never just which is best, but which is best for the money.
- No infrastructure to run
- Amber holds the provider keys, runs the sandboxes and meters every token. You use the web app, the CLI or the API.
How Amber differs
Other tools test your app. Amber tests the Model.
Each of these is good at what it does. None of them answers the question Amber exists for: which Model Configuration should we use for our work, and at what cost?
| Tool | Tests | Runs | Doesn’t answer |
|---|---|---|---|
| eve evals | Your own agent’s scripted behaviour | Your CI | Which Model, or how capable it is on real work |
| Braintrust, LangSmith, Langfuse | Outputs and traces, scored by judges | Your app | Whether a Model can finish real work in a real Environment |
| EvalRouter | Models on public, mostly static benchmarks | Their sandboxes; agents run on your machine | How Models rank on your agentic workload |
| Amber | Model Configurations doing real Tasks in Environments, on public and private Benchmarks | Fully hosted | — |
Pay for what the Trials cost. Plus 12.5%.
Model and sandbox costs pass through at cost, from a prepaid Wallet, and you see a Quote before anything is spent.