Skip to content
Hacker News front page

Snorkel releases Senior SWE-Bench, a benchmark that evaluates coding agents like senior engineers

Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

This benchmark gives agents natural-language feature requests and bug reports instead of over-specified specs. A validation agent writes behavioral tests to check solutions, and a taste-scoring system grades whether the code fits the codebase's actual style. The dataset and scoring approach are open-sourced on GitHub.

Why it matters: Snorkel's open-source Senior SWE-Bench swaps fixed spec documents for natural-language feature requests and bug reports, with a verifier agent writing behavior tests.

Read the original ↗Export Markdown