Benchmark
Benchmark
Benchmark
198 graduate-level biology, physics and chemistry questions written and validated by domain experts, chosen because experts agree on the answer and skilled non-experts with web access mostly get it wrong.
Expert-level scientific reasoning that can't be looked up.
How Amber’s Verifier and Rewards compare with the original’s.
Rein et al. (2023) score accuracy of the chosen option, zero-shot or few-shot, with and without chain of thought. This is zero-shot with a chain-of-thought request. The option order is the fixed shuffle of the public copy rather than a fresh shuffle per question, so Models are always shown the same order. The questions are from the official Diamond set, which asks that examples not appear in plain text online: do not copy these Instructions into public pages.
Amber static adapter: https://huggingface.co/datasets/fingertap/GPQA-Diamond/tree/68be7564497676e07a77a042fdb587deb88c51c3
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
198 Tasks · published 2026-10-21
17de37c12555bf3a3a3e5a15b661e7400cc0bb273e56fe73a23f2a24b04501cf68be7564497676e07a77a042fdb587deb88c51c3