Benchmark
Benchmark
Benchmark
1,319 grade-school maths word problems that need two to eight steps of arithmetic reasoning. Each has a single integer answer.
Multi-step arithmetic reasoning on everyday word problems.
How Amber’s Verifier and Rewards compare with the original’s.
Cobbe et al. (2021) score a problem as solved when the final numeric answer equals the reference after '####'. The numeric Verifier does the same, comparing exact values, so '18', '18.00' and '$18' agree and '1,234' equals 1234. It reads the `Answer:` line and, if a Model omits it, the last number in the response (as lm-evaluation-harness's flexible extraction does). Zero-shot with a chain-of-thought request, not the paper's fine-tuned-model or 8-shot setups.
Amber static adapter: https://huggingface.co/datasets/openai/gsm8k/tree/740312add88f781978c0658806c59bc2815b9866
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
1,319 Tasks · published 2026-10-21
a2e94100fe526024cdc63eb2a161ca733382aff01f78c6571eb38150c413ec75740312add88f781978c0658806c59bc2815b9866