Benchmark
Benchmark
Benchmark
6,508 questions from 27 subtasks of 23 challenging BIG-Bench tasks: logical deduction, causal judgement, word sorting, multi-step arithmetic, navigation and more. Earlier language models scored below the average human rater on them.
Algorithmic and multi-step reasoning across many small, varied puzzle types.
How Amber’s Verifier and Rewards compare with the original’s.
Suzgun et al. (2022) extract the text after 'So the answer is' from a three-shot chain-of-thought completion and score exact match, per subtask then averaged. Here Tasks are zero-shot with a chain-of-thought request, the answer is read from the `Answer:` line, and each Task earns a Reward of 0 or 1 (so the Benchmark's mean Reward is the micro-average over questions, not the subtask-level average). Option answers compare the chosen letter, numeric ones compare exact numbers, and the other subtasks compare text ignoring case and decoration (the Dyck-language answers also ignore spacing). Three of the 6,511 questions (movie_recommendation #163, ruin_names #99 and #144) are left out: the source split their options at commas, so no option is the answer.
Amber static adapter: https://huggingface.co/datasets/lukaemon/bbh/tree/982bb89fd79532a8ac676a61fc42eb1aeec63f99
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
6,508 Tasks · published 2026-10-21
74020158d424529b6577e4db0ec40b2b4132db09cdba13e7bb85ab4a0a2e5f28982bb89fd79532a8ac676a61fc42eb1aeec63f99