Catalog
Benchmarks
Public Benchmarks you can run today. Each Benchmark Version is an immutable snapshot of its Tasks, so Rewards stay comparable. Your own private Benchmarks appear only in your Workspace.
Catalog
Public Benchmarks you can run today. Each Benchmark Version is an immutable snapshot of its Tasks, so Rewards stay comparable. Your own private Benchmarks appear only in your Workspace.
The 30 problems from the 2025 AIME I and AIME II contests. Each has an integer answer from 0 to 999, and the problems were released after most Models' training data ended.
1,172 multiple-choice science questions from grade 3 to 9 standardised tests, restricted to those that simple retrieval and co-occurrence methods got wrong.
225 of Exercism's hardest programming exercises across C++, Go, Java, JavaScript, Python and Rust. The Agent implements each exercise and the provided unit tests decide whether it works.
6,508 questions from 27 subtasks of 23 challenging BIG-Bench tasks: logical deduction, causal judgement, word sorting, multi-step arithmetic, navigation and more. Earlier language models scored below the average human rater on them.
198 graduate-level biology, physics and chemistry questions written and validated by domain experts, chosen because experts agree on the answer and skilled non-experts with web access mostly get it wrong.
1,319 grade-school maths word problems that need two to eight steps of arithmetic reasoning. Each has a single integer answer.
10,042 short scenes from ActivityNet and WikiHow, each with four possible next sentences. The wrong endings were generated and filtered to fool language models while staying obvious to people.
541 prompts that each carry one to three instructions a program can check: length, keywords, formatting, case, punctuation, language and structure.
500 competition mathematics problems (AMC, AIME and similar) in seven subjects and five difficulty levels, a representative subset of the MATH test set.
12,032 college-level questions in 14 subjects, each with up to ten answer options. A harder, less guessable successor to MMLU.
500 real GitHub issues from popular Python projects, each confirmed solvable by human reviewers. The Agent edits the repository, and the project's own tests decide whether the issue is fixed.
Hard, realistic tasks in a Linux terminal: building and debugging software, training a model, configuring a server, analysing data. Each Task runs in its own container with tests that check the final state.