SWE-bench Verified

SWE-bench Verified is a benchmark in which a model has to fix real software bugs in popular open-source Python projects. The model gets a GitHub issue and the repository, and must change the code itself until the automated tests pass. The tasks were reviewed by OpenAI engineers, so a task is only in the set when it is clearly reproducible and solvable.

SWE-bench Verified measures how well a model works as an autonomous coding agent. In detail, it checks:

• finding the right file(s) in a large real codebase
• writing a fix that solves the described issue
• making sure the project's existing tests still pass
• working autonomously with terminal and file-editing tools

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the SWE-bench Verified benchmark?

SWE-bench Verified is a benchmark in which a model has to fix real software bugs in popular open-source Python projects. The model gets a GitHub issue and the repository, and must change the code itself until the automated tests pass. The tasks were reviewed by OpenAI engineers, so a task is only in the set when it is clearly reproducible and solvable.

What does SWE-bench Verified test?

SWE-bench Verified mainly measures how well a model works as an autonomous coding agent. Concretely, it checks: finding the right file(s) in a large real codebase; writing a fix that solves the described issue; making sure the project's existing tests still pass.

How is the SWE-bench Verified score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get SWE-bench Verified data and scores?

The official source is https://www.swebench.com/. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/swe-bench-verified/scores) or pull it directly from the Open LLM Radar API with any HTTP client.