SWE-bench Pro

SWE-bench Pro is a harder, cleaned-up variant of SWE-bench: the task set was revised to remove flawed tasks from the original collection so scores are less distorted. As with SWE-bench Verified, an agent solves real GitHub issues from real repositories under defined conditions (temperature and context window are fixed).

SWE-bench Pro measures how well a model works as an autonomous coding agent. In detail, it checks:

• real bug fixing across multiple programming languages
• large, realistic code repositories instead of toy projects
• correct multi-file, multi-step changes by an autonomous agent
• clean task definitions without ambiguous or broken test setups

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better.

What is the SWE-bench Pro benchmark?

SWE-bench Pro is a harder, cleaned-up variant of SWE-bench: the task set was revised to remove flawed tasks from the original collection so scores are less distorted. As with SWE-bench Verified, an agent solves real GitHub issues from real repositories under defined conditions (temperature and context window are fixed).

What does SWE-bench Pro test?

SWE-bench Pro mainly measures how well a model works as an autonomous coding agent. Concretely, it checks: real bug fixing across multiple programming languages; large, realistic code repositories instead of toy projects; correct multi-file, multi-step changes by an autonomous agent.

How is the SWE-bench Pro score calculated?

The score is accuracy: the percentage of tasks answered or solved correctly. 100% means every task was solved perfectly; higher is better. The declared maximum score of this benchmark is 100.0.

Where do I get SWE-bench Pro data and scores?

The official source is https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro. The official benchmark page/reference is linked on this page. Every score on Open LLM Radar is also available as open data: download the scores JSON (http://open-llm-radar-api:8000/api/v1/benchmarks/swe-bench-pro/scores) or pull it directly from the Open LLM Radar API with any HTTP client.