Snorkel releases Senior SWE-Bench, a benchmark that evaluates coding agents like senior engineers
Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
This benchmark gives agents natural-language feature requests and bug reports instead of over-specified specs. A validation agent writes behavioral tests to check solutions, and a taste-scoring system grades whether the code fits the codebase's actual style. The dataset and scoring approach are open-sourced on GitHub.
Why it matters: Snorkel's open-source Senior SWE-Bench swaps fixed spec documents for natural-language feature requests and bug reports, with a verifier agent writing behavior tests.