Benchmark
Benchmark
Benchmark
500 competition mathematics problems (AMC, AIME and similar) in seven subjects and five difficulty levels, a representative subset of the MATH test set.
Competition-level mathematical problem solving with a closed-form answer.
How Amber’s Verifier and Rewards compare with the original’s.
Hendrycks et al. (2021) take the content of the last \boxed{} and score exact match after normalising the LaTeX (fractions, units, degree signs, spaces). The exact Verifier uses that normalisation, plus unwrapping \text{...}, so 'Evelyn' matches '\text{Evelyn}'. It does not decide that two different expressions are equal (for example '0.25' and '\frac{1}{4}' are, but 'x+1' and '1+x' are not), which is the same limitation as the original. Zero-shot.
Amber static adapter: https://huggingface.co/datasets/HuggingFaceH4/MATH-500/tree/6e4ed1a2a79af7d8630a6b768ec859cb5af4d3be
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
500 Tasks · published 2026-10-21
519f0f03ed5e276454048f9b78b9d44765f29a840d531e243b9098772fcb80b56e4ed1a2a79af7d8630a6b768ec859cb5af4d3be