SWE-bench

How it works

SWE-bench is a family of benchmarks built from real issues and pull requests in open-source Python repositories. The model gets the repository and the issue text and must produce a code change; it is scored by whether the tests linked to the original fix now pass without breaking other tests.

There are several variants, including SWE-bench Verified, a human-checked subset of well-specified, solvable tasks, and others covering more languages or harder tasks. Scores depend heavily on the agent harness, tools, attempts and time allowed, which makers describe in their reports.

It is a useful signal for agentic coding on real codebases, but it covers specific repositories and kinds of tasks. Check the variant and harness behind a score before comparing.

Worked example

Illustrative: a 70% score on a 500-task variant means 350 tasks were resolved, i.e. the model's patch made the target tests pass without breaking the others. A score on a different variant with different tasks is not directly comparable.

Common mistakes

• 1Comparing a Verified score with a score on another SWE-bench variant.
• 2Ignoring the harness: the same model can score quite differently with different scaffolding.
• 3Assuming a high score means equal skill in your language, stack or repository size.