Benchmark
Benchmark
Benchmark
The 30 problems from the 2025 AIME I and AIME II contests. Each has an integer answer from 0 to 999, and the problems were released after most Models' training data ended.
Olympiad-style mathematical reasoning on problems unlikely to be in training data.
How Amber’s Verifier and Rewards compare with the original’s.
The AIME has no scoring paper: a problem is correct when the integer answer matches the official key, which is what the numeric Verifier checks, reading the last \boxed{} (or `Answer:` line, or failing that the last number in the response). The 30 problems are few, so a Model's result carries a wide confidence interval; run several Trials per Task. Zero-shot.
Amber static adapter: https://huggingface.co/datasets/opencompass/AIME2025/tree/a6ad95f611d72cf628a80b58bd0432ef6638f958
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
30 Tasks · published 2026-10-21
50de37b052cdb6ba639762478cab3b2dc25257f38c7b7a1e2c88db3b30bf0c9fa6ad95f611d72cf628a80b58bd0432ef6638f958