Benchmark
Benchmark
Benchmark
1,172 multiple-choice science questions from grade 3 to 9 standardised tests, restricted to those that simple retrieval and co-occurrence methods got wrong.
Grade-school science knowledge and commonsense reasoning.
How Amber’s Verifier and Rewards compare with the original’s.
Clark et al. (2018) score the fraction of questions answered correctly (with partial credit for ties) on the Challenge test set. This is accuracy of the chosen option with one Reward per question and no tie credit; Models answer in text with a chain-of-thought request, rather than by option likelihood as in lm-evaluation-harness. Questions labelled 1-4 in the source are presented as A-D.
Amber static adapter: https://huggingface.co/datasets/allenai/ai2_arc/tree/210d026faf9955653af8916fad021475a3f00453
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
1,172 Tasks · published 2026-10-21
f874c2a998bbe9510620e6d336b7ac6a9a1d2e6b2a38224c9497318f01a8dfc4210d026faf9955653af8916fad021475a3f00453