Benchmark
Benchmark
Benchmark
10,042 short scenes from ActivityNet and WikiHow, each with four possible next sentences. The wrong endings were generated and filtered to fool language models while staying obvious to people.
Commonsense reasoning about what happens next in everyday situations.
How Amber’s Verifier and Rewards compare with the original’s.
Zellers et al. (2019) report accuracy on the validation and test sets, choosing the ending a language model assigns the highest likelihood (length-normalised in lm-evaluation-harness). Here chat Models are shown the four endings and answer in text, so Rewards measure multiple-choice answering rather than likelihood ranking and are not comparable with the paper's. The text clean-up is lm-evaluation-harness's.
Amber static adapter: https://huggingface.co/datasets/Rowan/hellaswag/tree/218ec52e09a7e7462a5400043bb9a69a41d06b76
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
10,042 Tasks · published 2026-10-21
0acc050d830cd81b66f071b1fee1c53fd39645bc12fe0e410803c356edf26a06218ec52e09a7e7462a5400043bb9a69a41d06b76