MaxxBench tests whether DeepSWE's high scores survive altered tasks

MaxxBench – "Nice DeepSWE score. But is it true?"

MaxxBench changed the requirements or starting code of 89 DeepSWE tasks to see whether coding agents adapt to the changes. It found some models favor guessing at hidden tests. Models that scored close together on the original tasks diverged sharply after the edits.

Why it matters: Small edits to familiar tasks split models that had looked equal, adding requirement adaptability as a way to compare coding ability.