After SWE-Bench Pro Saturation, Someone Built a New Benchmark
SWE-Bench Pro 饱和之后,有人做了一把新尺子
DeepSWE says SWE-Bench Pro lost discrimination because of data contamination and verifier flaws; the same model set showed a 62-point spread on the benchmark, while the snippet does not disclose the audited models or test protocol.
Why it matters: Datacurve's DeepSWE benchmark spreads the same models across a 62-point gap where SWE-Bench Pro shows only a few points, showing the old benchmark can no longer separate models.