Benchmark
Benchmark
Benchmark
541 prompts that each carry one to three instructions a program can check: length, keywords, formatting, case, punctuation, language and structure.
Whether a Model follows precise, verifiable formatting and content instructions.
How Amber’s Verifier and Rewards compare with the original’s.
Prompt-level strict accuracy as in Zhou et al. (2023): every instruction must be satisfied and the response is not rewritten before checking. NLTK's sentence and word tokenisers are replaced by regular expressions, and langdetect is seeded for repeatable results. The dataset ships a single split, named 'train' on Hugging Face.
Amber static adapter: https://huggingface.co/datasets/google/IFEval/tree/966cd89545d6b6acfd7638bc708b98261ca58e84
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
541 Tasks · published 2026-10-21
36d9ec4a4e81cbb2bd323bc301034dbccf3376e7da542b12a1facd16763352f1966cd89545d6b6acfd7638bc708b98261ca58e84