SWE-bench Pro
SWE-bench Pro - refined benchmark with corrected problematic tasks in the public set. Internal agent scaffold; temp=1.0, top_p=0.95, 200K context window.
Category
Agentic Coding
Max Score
100.0
Score Type
percent
Active
Yes
Model Scores
Scores for SWE-bench Pro
| Model | Score | Date | Verified |
|---|---|---|---|
| Qwen3.6-35B-A3B |
61.88%
|
16.04.2026 | Verified |
| GPT 5.5 |
73.25%
|
11.06.2026 | Unverified |
| Kimi K2.6 |
73.25%
|
11.06.2026 | Unverified |
| MiniMax-M2.7 |
70.25%
|
11.06.2026 | Unverified |
| Gemma4-31B-it |
44.62%
|
16.04.2026 | Unverified |
| Qwen3.5-35B-A3B |
55.75%
|
16.04.2026 | Unverified |
| Gemma4-26B-A4B |
17.25%
|
16.04.2026 | Unverified |
| Qwen3.6-27B |
66.88%
|
22.04.2026 | Verified |
| Qwen3.5-397B-A17B |
63.62%
|
22.04.2026 | Unverified |
| Claude 4.5 Opus |
71.38%
|
22.04.2026 | Unverified |
| Qwen3.5-27B |
64.00%
|
16.04.2026 | Unverified |
| Ornith-1.0-9B |
53.62%
|
25.06.2026 | Verified |
| Ornith-1.0-35B |
63.00%
|
25.06.2026 | Verified |
| Qwen3.8-27B |
77.12%
|
18.08.2026 | Verified |
| Qwen3.7-Plus |
72.00%
|
18.08.2026 | Unverified |
| Muse Glimmer-30B |
64.00%
|
18.08.2026 | Unverified |
| Opus4.6 Max |
66.75%
|
18.08.2026 | Unverified |
| Qwen3.8-2.4T-A95B |
84.62%
|
18.08.2026 | Verified |
| Fable-5 |
100.00%
|
18.08.2026 | Unverified |
| GPT-5.6 Sol |
80.75%
|
18.08.2026 | Unverified |
| Claude Opus 4.8 |
86.50%
|
18.08.2026 | Unverified |
| Muse-Glimmer-30B |
64.00%
|
01.08.2026 | Verified |
| DeepSeek V4 Pro |
69.25%
|
11.06.2026 | Unverified |
| Claude Opus 4.7 |
80.38%
|
11.06.2026 | Unverified |
| MiniMax-M3 |
73.75%
|
11.06.2026 | Verified |
| Gemini 3.1 Pro |
67.75%
|
11.06.2026 | Unverified |
| GLM-5.2 |
77.62%
|
17.06.2026 | Verified |
| DeepSeek-V4-Pro |
69.25%
|
17.06.2026 | Unverified |
| GLM-5.1 |
73.00%
|
17.06.2026 | Unverified |
| GPT-5.5 |
73.25%
|
17.06.2026 | Unverified |
| Qwen3.7-Max |
75.75%
|
17.06.2026 | Unverified |
| Qwen3-Coder-Next |
55.38%
|
03.02.2026 | Verified |
| DeepSeek-V3.2 |
51.12%
|
03.02.2026 | Verified |
| GLM-4.7 |
50.75%
|
03.02.2026 | Verified |
| MiniMax M2.1 |
43.25%
|
03.02.2026 | Verified |
| Claude-Opus-4.5 |
58.75%
|
03.02.2026 | Verified |
| Claude-Sonnet-4.5 |
55.00%
|
03.02.2026 | Verified |
| Kimi K2.5 |
50.00%
|
03.02.2026 | Verified |
| Ornith-1.5-35B-A3B |
74.50%
|
19.08.2026 | Verified |
| Ornith-1.0-35B-A3B |
63.00%
|
19.08.2026 | Unverified |
| Gemma-4-31B |
44.62%
|
19.08.2026 | Unverified |
| GPT-5.4 |
72.12%
|
23.08.2026 | Verified |
| Claude Opus 4.6 |
66.75%
|
23.08.2026 | Verified |
| Opus-4.6 |
71.62%
|
26.06.2026 | Verified |
| DeepSeek-V4-Pro |
69.25%
|
26.06.2026 | Verified |
| DeepSeek-V4-Flash |
65.75%
|
26.06.2026 | Verified |
| Gemma 3 27B IT |
14.22%
|
27.02.2025 | Verified |
| Qwen3.8-Flash-Next |
78.12%
|
26.08.2026 | Verified |
| DeepSeek-V4-Flash-0731 |
70.00%
|
26.08.2026 | Unverified |
| Claude-Opus-4.6 (Max) |
66.75%
|
26.08.2026 | Unverified |
| Hy3 |
72.38%
|
28.08.2026 | Verified |
| Hy4 preview |
82.12%
|
28.08.2026 | Verified |
| GLM-5.3 |
80.75%
|
28.08.2026 | Verified |
| Kimi K3 |
79.12%
|
28.08.2026 | Verified |
| Claude Opus 5 |
99.00%
|
28.08.2026 | Verified |
| Spark-X2.5-4B |
55.50%
|
— | Verified |
| Spark-X2.5-1.7B |
13.00%
|
— | Verified |
| Qwen3.5-4B |
36.75%
|
— | Verified |
| Qwen3.5-2B |
2.38%
|
— | Verified |
| Gemma4-E4B |
5.00%
|
— | Verified |
| Qwen3.5-9B |
42.25%
|
25.06.2026 | Verified |
| Gemma4-12B |
27.38%
|
25.06.2026 | Verified |
| MiniCPM5-2B |
18.00%
|
— | Verified |
| LFM2.5-2.6B |
0.75%
|
— | Verified |
| Gemma4-E2B |
0.00
|
— | Verified |
| granite-4.2-3B |
15.38%
|
— | Verified |
| Nemotron-3-Nano-4B |
0.12%
|
— | Verified |
| LFM2.5-8B-A1B |
0.50%
|
— | Verified |
| GLM 5.1 Thinking |
73.00%
|
11.06.2026 | Unverified |