The Benchmarks Are Lying: Why SWE-bench Verified and MMLU Can't Tell You Which Model Is Better Anymore
Claude Opus scores 80.9% on SWE-bench Verified. The same model, on the harder SWE-bench Pro variant, scores 45.9%. That is not two different capability levels, it is one model and a 35-point gap that exists purely because of which test you ran. If a benchmark number can swing that much depending on which version of the same eval you pick, the number was never a stable measurement to begin with.
Contamination is not a conspiracy theory anymore
OpenAI's own Frontier Evals team has documented models reproducing the original gold patch or problem statement from an evaluation verbatim, with minimal prompting, using nothing but the task ID. That is not reasoning its way to an answer, that is recall. SWE-bench Verified has been public long enough, and referenced in enough training-adjacent web content, that a meaningful share of its problems are now more memorized than solved. Over 60% of the problems models still fail on are considered structurally broken tests, too narrow or too loosely specified to grade fairly, which tells you the remaining "hard" cases aren't even clean signal either.
MMLU and GPQA Diamond have a different problem: saturation. Frontier models now cluster at around 94% on GPQA Diamond, against a PhD-expert human baseline of roughly 65%. When every top model lands within a point of every other top model, the benchmark has stopped separating anything. A leaderboard where first place and fourth place are statistically indistinguishable isn't ranking capability, it's rounding error with a scoreboard attached.
What's actually still discriminative
Not every benchmark has this problem. LiveBench, an ICLR 2025 Spotlight paper, refreshes its question set monthly specifically to stay ahead of contamination, and top models currently score below 70% on it, well under the 90%+ they post on saturated statics like MMLU. SWE-bench Pro, the harder sibling of SWE-bench Verified, produces a real spread between models instead of a clustered tie. The pattern across both: benchmarks built to resist memorization and update on a real cadence still separate models meaningfully. Static benchmarks that have been public for a year or more, and that get quoted heavily in marketing copy, increasingly don't.
How to actually read a benchmark claim
- Ask which version of the benchmark the number comes from. "SWE-bench" alone is ambiguous now, Verified and Pro produce meaningfully different numbers on the same model.
- Treat any benchmark where top models cluster within a couple points of each other as a tie, not a ranking, regardless of what order the leaderboard displays them in.
- Favor benchmarks that refresh on a schedule (LiveBench-style) over static ones that have been public long enough to leak into training data.
- Run the tool against your own actual workload before trusting a leaderboard position to predict how it'll perform on your codebase.
None of this means benchmarks are worthless. It means the specific number on a vendor's homepage is doing less work than it looks like it's doing, and the gap between a saturated benchmark and a contamination-resistant one is exactly where the real signal still lives.