1Password's FLAWED paper on AI patching criticized for thin citations and factual errors

FLAWED's Flaws and What This Means for Industry Research

Suha Sabi Hussain publicly criticized 1Password's FLAWED paper from Off-by-1 Labs. The paper claims frontier models often produce flawed vulnerability patches, but Hussain notes it cites only 19 sources—mostly corporate blogs and XKCD—while omitting directly relevant prior work like Meta's AutoPatchBench and an NDSS paper. The paper also contains mislabeled diagrams and arithmetic errors. Hussain argues that 1Password adopted the tone of rigorous research without the corresponding rigor, and that this work overshadowed higher-quality research from less-resourced groups like EleutherAI. She calls for a retraction or correction and suggests partnering with academic researchers.

Why it matters: The author, a security researcher, provides concrete evidence (missing citations to Meta's AutoPatchBench and an NDSS paper) against 1Password's FLAWED paper — not empty criticism. But it's a personal blog rebuttal, not primary research or a product launch, so importance sits ...