Harbor Task format

A private Benchmark is a set of Tasks in Harbor's Task format. You upload one or more Task folders as a .tar.gz, .tar or .zip (up to 100 MB compressed, 1 GB and 50,000 files unpacked, 1,000 Tasks), and Amber validates them before they can be used.

#Folder layout

Each Task is a folder:

my-task/
├── task.toml          # metadata and limits
├── instruction.md     # the Instruction given to the Agent
├── environment/
│   └── Dockerfile     # builds the Environment the Agent works in
├── solution/
│   └── solve.sh       # a reference solution
└── tests/
    └── test.sh        # the Verifier

Upload a folder that contains several Task folders to create a Benchmark with several Tasks. Every folder with a task.toml is a Task. If task.toml has a [task] section, its name is org/name and the Task is called by the part after the slash; otherwise by its folder's name. Names use letters, digits, ., _ and -, and are unique in an upload.

#What an archive may hold

Uploads are untrusted, so Amber unpacks them with strict rules and rejects the whole archive if any entry breaks one:

• only files and folders: no symbolic or hard links, devices or other special files, and no encrypted zip entries;
• relative paths with forward slashes: no absolute paths, no .., no backslashes;
• every COPY and ADD in a Dockerfile reads from inside the Dockerfile's own folder (environment/), never from outside the Task folder.

And each Task's task.toml must not:

• fill an env value ([environment.env], [solution.env], [verifier.env], a step's or a separate Verifier Environment's) from the machine running it with ${VAR} or ${VAR:-default}: give literal values;
• ask for GPUs or TPUs (gpus, gpu_types, tpu);
• ask for more than one sandbox has: 4 cpus, 8192 memory_mb and 10240 storage_mb.

A Task that breaks one fails its format check; the rest of the upload goes on.

File permissions are kept only as far as "executable or not".

#The Verifier

tests/test.sh runs after the Agent finishes, inside the same Environment. It inspects the end state and writes a Reward between 0 and 1. Write the check against the outcome (a file exists, a test suite passes, a service responds) and not against how the Agent got there.

#Validation

Uploading runs validation, and every check must pass for a Task to be included:

1. The format is valid.
2. The Environment builds.
3. The oracle Agent, which runs the Task's own solution, gets a Reward of 1.
4. The nop Agent, which does nothing, gets a Reward of 0.

Validation runs are billed to your Wallet: one oracle and one nop Trial per Task, which call no Model, so you pay for the Environments' time. Amber shows a Quote first. Your Tasks are your own, so a validation Trial whose Environment doesn't build or whose Verifier crashes or times out is charged and fails that Task, and a validation Trial is stopped after 1.5 times the sandbox time its Quote expected. Failures are reported per Task, and you can exclude failing Tasks and publish the rest. Publishing creates a new private Benchmark Version.

An upload that is checked but never validated keeps its Tasks for 7 days; one that was validated but not published keeps them for 30 days.

#Good Tasks

• One clear outcome. The Instruction says what to achieve, not how.
• Deterministic checks. The Verifier gives the same Reward for the same end state.
• No network. While the Agent works (and while the oracle Agent runs your solution), Environments have no network access except the Amber model proxy and the hosts you list when you upload (the network allowlist). Bake dependencies into the image.
• Hard to game. A nop Agent must get a Reward of 0, so the starting state must not already satisfy the check.

#Privacy

A private Benchmark belongs to one Workspace. It is never visible outside it, in the catalog, in search, or in errors. Uploaded Tasks are untrusted code and run only in isolated sandboxes. You can restrict which Providers your private Benchmarks may be sent to in your Workspace's provider policy.