Benchmark
Benchmark
Benchmark
Hard, realistic tasks in a Linux terminal: building and debugging software, training a model, configuring a server, analysing data. Each Task runs in its own container with tests that check the final state.
Whether an Agent can finish long, multi-step work in a real terminal Environment.
How Amber’s Verifier and Rewards compare with the original’s.
Terminal-Bench runs Agents with the whole internet. Under Amber an Agent reaches only the Amber model proxy and package mirrors (Ubuntu and Debian apt, PyPI, PyTorch's CPU wheels, CRAN); each Task's Verifier keeps the network access the Task declares. Tasks whose instructions send the Agent to other sites (GitHub, Hugging Face, project download pages) cannot be done the way their authors intended, so scores can be lower than on the original. Version 2.0 leaves out three Tasks it can't run: mteb-leaderboard and mteb-retrieve, whose 2.0 images unpack to about 11.8 GiB, more than the 10 GiB a sandbox has here (2.1 rebuilt them at under 1 GiB), and extract-moves-from-video, which asks the Agent to download a YouTube video (2.1 ships the video with the Task).
Harbor registry: https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
89 Tasks · published 2026-10-21
97f6b88232e9d70187ba483ae076ade48145e8ddc2be74009495a8384887decdsha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a86 Tasks · published 2026-10-21
ef0f4df6f8ca9e920bff204a3182820b8b451c52ec784a6e175e6f58fa58e920sha256:c6fc2e2382c1dbae99b2d5ecd2f4f4a60c3c01e0d84642d69b4afd92e99d078b