Skip to content
Hacker News front page

Show HN: I benchmarked LLM agents on fixing real-world security vulnerabilities

Giovanni Gatti benchmarked 5 LLM agents on 20 real CVEs across 18 Python projects, and the best solve rate across 300 runs was 50%.

Why it matters: The author tested 5 frontier models on 20 real Python vulnerabilities, and across 300 runs the best fixed only half, with the priciest and cheapest models performing similarly at a 12x cost gap.

Read the original ↗Export Markdown