Benchmark
Benchmark
Benchmark
225 of Exercism's hardest programming exercises across C++, Go, Java, JavaScript, Python and Rust. The Agent implements each exercise and the provided unit tests decide whether it works.
Writing and editing code correctly across six programming languages.
How Amber’s Verifier and Rewards compare with the original’s.
Agents reach only the Amber model proxy and the package registries for npm, Cargo and PyPI, so the JavaScript, Rust and Python exercises can install what their tests need. The 47 Java exercises run their tests with the Gradle wrapper, which downloads Gradle from GitHub, and Amber doesn't allow that: an Agent can compile with javac but not run the Gradle tests, which the Verifier still runs with the network access the Task declares.
Harbor registry: https://hub.harborframework.com/datasets/aider/aider-polyglot
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
225 Tasks · published 2026-10-21
a7268e0d058432e2796f7fd37e37dfb3b031d60aee01de4473773c7bcbf6b193sha256:01e28d85e46beae5b7e29a29f57cb49d882b5486583d52cec4ee5bf3540a1c84