Documentation

Amber answers one question: which Model Configuration should we use for our work, and at what cost? It runs Model Configurations on Tasks, in real sandboxed Environments, on public and private Benchmarks, and shows the Reward each one earns and what it costs.

#Where to start

• Quickstart: install the CLI, get a Quote and compare two Model Configurations.
• Concepts: the vocabulary used everywhere in Amber, defined once.
• CLI reference and API reference: every command and endpoint.
• Harbor Task format and Guided authoring: measure your own work with a private Benchmark.
• FAQ: cost, credit, privacy and how Amber differs from other tools.

#How the pieces fit

You choose one or more Benchmark Versions, one or more Agents and one or more Model Configurations. Amber turns that Job into Trials, one for every Task, Agent and Model Configuration combination (times the number of Trials per Task you asked for). Each Trial runs in its own Environment, and the Task's Verifier produces a Reward. Amber then compares the Rewards, with confidence intervals, next to what each Model Configuration cost.