Docs
FAQ
Answers about cost, credit, privacy, comparisons and how Amber differs from other tools.
#Cost
#What does Amber cost?
The pass-through cost of Models and sandboxes, plus a 12.5% platform fee, from a prepaid Wallet. See pricing for a worked example.
#Do I get free credit when I sign up?
No. A new Workspace starts with an empty Wallet. You can browse Benchmarks and get Quotes straight away; to run a Job, the Wallet needs credit.
#Can I add money to my Wallet?
Top-ups are coming soon. For now, contact us with your Workspace's name and we add credit to its Wallet by hand.
#Can a Job spend more than its Quote?
No new Trials start once the Quote is reached. Trials already in flight finish, and any overshoot from them is absorbed by Amber, not charged to you.
#Am I charged for Errored Trials?
No. They are retried up to two more times, and if they still error they are left out of the averages and not charged. On a private Benchmark, a failure of your own Task (its Environment doesn't build, its Verifier crashes or times out, or the Trial runs past its time limit) is not Errored: it is a Reward of 0 and is charged.
#Results
#Why does Amber say "too close to call"?
Because the 95% confidence interval of the paired difference between two Model Configurations contains zero: with these Tasks and Trials, the difference could be noise. You can request a Quote for extra Trials on just those cells.
#Why can't I compare results from different Agents?
Rewards are comparable only when both the Benchmark Version and the Agent Version match. Anything else is shown on a separate panel with an explanation.
#How long are transcripts kept?
90 days, then they are deleted. Rewards, costs and aggregates are kept. You can delete a Job's transcripts earlier from your Workspace.
#Privacy
#Who can see my private Benchmarks?
Only the members of your Workspace. Amber staff can see them only if you turned on staff access for done-for-you work, and those actions are audit-logged.
#Where are my Tasks sent?
To the Providers that run the Models in your Jobs. You can restrict this to an allowlist of Providers in your Workspace's provider policy, and OpenRouter requests for private Benchmarks ask not to collect data. See the privacy policy draft.
#Comparisons with other tools
#How is Amber different from eve, Braintrust or EvalRouter?
eve evals test your own agent's scripted behaviour in your CI. Braintrust, LangSmith and Langfuse judge outputs and traces from your app. EvalRouter measures Models on public, mostly static benchmarks, with agents running on your machine. Amber runs Model Configurations on real Tasks in hosted Environments, on public and private Benchmarks. The landing page has the full comparison.