Concepts

Amber uses a small, precise vocabulary. It follows Harbor's terms wherever Harbor has one, except that Amber says Benchmark for what Harbor calls a dataset.

#What gets evaluated

#Task

One unit of work for an Agent: an Instruction, an optional Environment to work in, and a Verifier that judges the result. A Task is either agentic (it has an Environment) or static (it has none; a single Model call is checked by a built-in Verifier).

#Instruction

The natural-language goal given to the Agent at the start of a Task.

#Environment

The isolated machine state a Task starts from and the Agent acts within. Environments have no network access by default, apart from the Amber model proxy and anything a Benchmark Version explicitly allows.

#Verifier

The part of a Task that inspects the Environment after the Agent finishes and produces a Reward. It looks at the end state, not at what the Agent said.

#Benchmark

A named collection of Tasks that is run and reported on as one whole. Benchmarks are either public (from Harbor's registry or Amber's static adapters) or private to a Workspace.

#Benchmark Version

An immutable snapshot of a Benchmark's Tasks. It is identified by a content hash and has a human label such as terminal-bench@2.0. Every Reward belongs to exactly one Benchmark Version, and any change to the Tasks makes a new one.

#Who gets evaluated

#Model

A specific language model from a specific lab, the thing you are choosing between.

#Provider

The company whose servers actually run a Model for a call, such as Anthropic directly, or a specific host reached through OpenRouter.

#Model Configuration

A Model together with its pinned Provider and the settings it is called with, such as reasoning effort and temperature. This is what you compare and choose between. For OpenRouter, the upstream host and quantization are pinned and fallbacks are off; if a Trial is served by a different Provider than the pinned one, it is Errored.

#Agent

The software that drives a Model through a Task by reading the Instruction and acting in the Environment. Amber ships Terminus-2 (the default), pi, mini-swe-agent, Claude Code (Anthropic Models only) and Codex CLI (OpenAI Models only).

#Direct Agent

The trivial Agent that makes a single Model call with the Instruction and no tools. It is used for static Tasks, and only for them.

#Agent Version

An immutable snapshot of an Agent's code and configuration. Rewards are comparable only when both their Benchmark Version and their Agent Version match.

#Running and results

#Job

A single request to run one or more Benchmarks against one or more Agent and Model Configuration pairs. A Job moves through quoted, running and then ends as completed, cancelled, ceiling_reached or failed.

#Trial

One attempt by one Agent and Model Configuration pair at one Task, within a Job. You can ask for 1 to 10 Trials per Task.

#Reward

The numeric result a Verifier assigns to a Trial, a number from 0 to 1. An Agent that times out or gives up gets a Reward of 0; that is a result, not a failure of the platform.

#Errored

The outcome of a Trial that could not produce a trustworthy Reward because of a failure outside the Model and Agent, such as the platform, the sandbox or the Provider. Errored Trials are retried up to two more times. If they still error, they have no Reward, are left out of the averages, are shown separately and are not charged.

On a private Benchmark the Tasks are yours, so a failure of the Task itself is not Errored: if its Environment doesn't build, the Agent can't be set up in it, its Verifier crashes, times out or writes no valid Reward, or the Trial runs past its time limit, the Trial gets a Reward of 0 and is charged, without retries.

#Ownership and money

#Workspace

The boundary that owns a Wallet, private Benchmarks and Jobs, and whose members alone can see them.

#Wallet

A Workspace's prepaid credit balance, which Jobs draw down as they spend on Models and Environments. It is an append-only ledger: an entry for every credit, reservation, charge and refund. A new Workspace's Wallet starts empty.

#Credit

Money Amber staff add to a Wallet by hand, with a reason, as opposed to a top-up you paid for. Until top-ups launch, it is how a Wallet gets money: contact us and we add it.

#Quote

The estimated cost of a Job, set aside from the Wallet before the Job starts and acting as its spending ceiling. A Quote expires after one hour. It is calculated per cell (Benchmark Version, Agent Version and Model Configuration) as Tasks times Trials per Task times the estimated cost per Trial.

#How results are compared

Amber reports, for each cell, the mean Reward (averaged per Task, then across Tasks) with a 95% bootstrap confidence interval resampled over Tasks, the cost, the mean duration and the number of Errored Trials.

Two Model Configurations in the same comparison get a paired bootstrap of the difference over the Tasks they share. If that interval contains zero, Amber shows too close to call.

Transcripts of each Trial are kept for 90 days and then deleted. Rewards, costs and aggregates are kept.