Skip to content
Hacker News front page

1Password's FLAWED paper on AI patching criticized for thin citations and factual errors

FLAWED's Flaws and What This Means for Industry Research

Suha Sabi Hussain publicly criticized 1Password's FLAWED paper from Off-by-1 Labs. The paper claims frontier models often produce flawed vulnerability patches, but Hussain notes it cites only 19 sources—mostly corporate blogs and XKCD—while omitting directly relevant prior work like Meta's AutoPatchBench and an NDSS paper. The paper also contains mislabeled diagrams and arithmetic errors. Hussain argues that 1Password adopted the tone of rigorous research without the corresponding rigor, and that this work overshadowed higher-quality research from less-resourced groups like EleutherAI. She calls for a retraction or correction and suggests partnering with academic researchers.

Why it matters: Security researcher Suha Sabi Hussain lists specific chart, arithmetic and citation errors in 1Password's FLAWED study and notes it misses direct work like Meta's AutoPatchBench.

Read the original ↗Export Markdown