Benchmark
Benchmark
Benchmark
12,032 college-level questions in 14 subjects, each with up to ten answer options. A harder, less guessable successor to MMLU.
Academic knowledge and reasoning across 14 subjects, with ten options per question.
How Amber’s Verifier and Rewards compare with the original’s.
Wang et al. (2024) report accuracy of the chosen option, with a five-shot chain-of-thought prompt and an answer extracted from 'the answer is (X)'. This is zero-shot with the same chain-of-thought request, reading `Answer: X`; the paper falls back to a random option when none is found, whereas a missing or invalid answer here earns a Reward of 0. Rewards are therefore not directly comparable with the leaderboard's five-shot numbers.
Amber static adapter: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro/tree/b189ec765aa7ed75c8acfea42df31fdae71f97be
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
12,032 Tasks · published 2026-10-21
c6d6f505a4de8b8f587f0bf92b989172bc46749602c853386ba26a6dceadc127b189ec765aa7ed75c8acfea42df31fdae71f97be