SWE-Bench Pro is broken. DeepSWE shows what an honest coding benchmark looks like
Why the old coding leaderboards produce nonsense and a new benchmark suddenly reveals massive gaps between models
Benchmarks are supposed to tell us which model actually codes well. For months now, that is exactly what they have failed to do. SWE-Bench Pro measures garbage, and the new DeepSWE benchmark reveals just how extensive the damage is.
The idea behind SWE-Bench Pro was good: realistic coding problems, real repositories, tough tasks. The problem is the execution. The tasks come from existing commits and pull requests, meaning solutions, tests, and discussions have long been available online. Models have seen them during training. They aren't guessing; they are remembering.
Worse still is the evaluation. A DataCurve audit found that the SWE-Bench Pro verifier produces around 8 percent false positives and 24 percent false negatives. In roughly 13 percent of the Opus 4.6 and 4.7 runs, the model cheated, and in 87 percent of those cases, the agent simply read the correct solution directly from the Git history. A benchmark that fails to catch this no longer measures anything useful.
And then there are the prompts. SWE-Bench Pro appends a system prompt that forbids the model from writing its own tests and dictates how to proceed in 15 steps. No human works with an agent like that. That single line—the instruction not to touch any tests—undermines its entire validity.
DeepSWE does things differently. Every task is written completely from scratch, not copied from a public commit, and no task is ever merged back into the original repositories. This eliminates contamination. On top of that, it includes 113 tasks across 91 active repositories in five languages: TypeScript, Go, Python, JavaScript, and Rust. SWE-Bench Pro publicly covers a mere 11 repos.
Short, realistic prompts
DeepSWE prompts are half the length of those in SWE-Bench Pro. They describe the behavior, not the implementation path. The agent has to figure out on its own where and how to build the change.
More real work
Despite shorter prompts, the solutions require 5.5 times more code and roughly twice as many output tokens. On average, 668 new lines across 7 files instead of 120 lines across 5.
Handwritten verifiers
The verifiers check observable behavior, not a specific implementation. Any sensible solution passes. False positive rate: 0.3 percent. False negative rate: 1.1 percent.
The result is a leaderboard that finally shows meaningful separation. On SWE-Bench Pro, many models sit within a narrow band. On DeepSWE, they spread across 70 percentage points.
| Model | DeepSWE | SWE-Bench Pro |
| GPT-5.5 | 70% | 59% |
| GPT-5.4 | 56% | 58% |
| Claude Opus 4.7 | 54% | 64% |
| Claude Sonnet 4.6 | 32% | 54% |
| Gemini 3.5 Flash | 28% | not comparable |
The drop between Opus 4.7 and Sonnet 4.6 is striking: 54 percent versus 32 percent, nearly cut in half. And Gemini 3.5 Flash lands at 28 percent, far from the top rather than within striking distance as shown on the old leaderboards. Anyone using these models for real work will recognize precisely these gaps.
One detail explains a lot: strong models test their own code voluntarily, as long as you don't forbid them from doing so. On DeepSWE, GPT-5.4 and Opus 4.7 write their own tests in over 80 percent of runs. On SWE-Bench Pro, where the prompt prohibits it, all models hover between 3 and 28 percent. The benchmark artificially kept the models dumb.
Things get interesting regarding costs and tokens, because the results finally mean something. GPT-5.5 hits 70 percent using around 47,000 output tokens at roughly $5.80 per run. Opus burns nearly twice as many tokens and costs around $16, with a lower score. Gemini 3.5 Flash, ostensibly the cheap, fast model, requires 150,000 tokens, ends up costing almost the same, and delivers a third of the performance. More tokens, more time, and more money do not correlate with better results.
For open-weight models, the picture is bleak. None reaches even half the score of the latest SOTA generation. Kimi K2.6 sits roughly at the level of GPT-5.4-mini, and that is not a good model. If you unleash an open-weight model in isolation on a small single file, it gets by. In a real codebase, it falls behind immediately.
DeepSWE is not perfect, and the team admits it openly. All models run via mini-swe-agent with only a single bash tool, which can disadvantage models trained on custom edit primitives. The repositories need at least 500 GitHub stars, leaving niche projects and proprietary code out of scope. Bug localization and refactoring are underrepresented, and languages like C++ and Java are currently missing.
Key Takeaways
Never take a single benchmark as gospel, especially not a contaminated one. DeepSWE is the most honest attempt yet to measure coding agents on real work, and the gap between the top tier and the rest is larger than the old rankings led us to believe. My advice remains simple: keep a log of your own failure cases, build a small private benchmark out of real tasks from your codebase, and test the finalists there before committing to a model.