Docs

Quickstart

Install the CLI, get a Quote and compare two Model Configurations in about ten minutes.

This walkthrough takes about ten minutes. You will install the CLI, sign in, ask for a Quote and then compare two Model Configurations on one Benchmark. Everything here is also available in the web app and the API.

Sign in with GitHub or Google. Amber creates a personal Workspace for you, with an empty Wallet. Top-ups are coming soon; until then, contact us with your Workspace's name and we add credit to it. You do not need a card, and you can follow steps 2 to 4 while you wait.

#2. Install the CLI

uv tool install amber-cli

Until the package is published to PyPI, install it from the repository:

uv tool install "git+https://github.com/CoreDepth/evalbench#subdirectory=py/amber_cli"

Then sign in. The CLI shows a short code and opens a page where you approve it; Amber issues an API key that the CLI stores.

amber login
amber whoami

#3. Pick a Benchmark and Model Configurations

Browse the Benchmarks and Models in the catalog, or list them from the terminal:

amber benchmarks list
amber models configs list

A good first choice is a small Benchmark with two Model Configurations you are already deciding between, for example opus-high and gpt-high.

#4. Get a Quote

A Quote is the estimated cost of a Job, shown before anything is spent.

amber quote -b terminal-bench@2.0 -m opus-high -m gpt-high

The Quote lists its breakdown (model, sandbox, judge and platform fee) and how confident the estimate is. If the Benchmark has been run before, it is based on real Trials; otherwise it comes from a prior and says so. A 20% safety margin is included.

#5. Run the Job

amber run -b terminal-bench@2.0 -m opus-high -m gpt-high

Amber shows the Quote again and asks you to confirm (add --yes to skip the question). Progress streams live. You can leave and come back with amber jobs wait <id>.

#6. Compare

amber results <job-id>
amber compare <job-id>

Results are the mean Reward per Model Configuration with a 95% confidence interval, the cost per Trial and any Errored Trials, which are shown separately and not charged. When two Model Configurations cannot be told apart, Amber says too close to call and offers a Quote for extra Trials on just those cells.

To keep the evidence, export it:

amber export <job-id> --format html -o report.html

#Next

Learn the concepts behind these terms.
Measure your own work with a private Benchmark.