Benchmark
Benchmark
Benchmark
500 real GitHub issues from popular Python projects, each confirmed solvable by human reviewers. The Agent edits the repository, and the project's own tests decide whether the issue is fixed.
Resolving real software issues in a real repository.
Harbor registry: https://hub.harborframework.com/datasets/swe-bench/swe-bench-verified
A Benchmark Version never changes. Any change to its Tasks makes a new one, identified by a content hash and pinned to its source revision.
500 Tasks · published 2026-10-21
8106e832edd3f03c49467cfcca5b7917863d0e823ee6236e51379b98bf093664sha256:b934b0cc3dc800fe945eaf9f1623329db97ee3133c706d20644524c7759fb341