Trail of Bits calls 1Password's AI patching benchmark misleading, releases two agent skills for patch validation

Trail of Bits 批评 1Password 的 AI 补丁基准存在误导，并发布两个补丁验证 Agent 技能

Trail of Bits reanalyzed 1Password's FLAWED report and argues the 26% clean-fix headline is misleading. Trials that deliberately instructed agents to apply wrong fixes or prohibited testing were mixed into the average. When restricted to trials where agents could run code and weren't given bad advice, 86% of patches blocked the supplied exploit. Trail of Bits also released two agent skills: post-patch-validation for automated patch testing, and review-walkthrough for engineer review. The post doesn't disclose performance data for these skills.

Why it matters: Trail of Bits re-analyzed 1Password's report claiming AI fixes only 26% of bugs, showing the headline number mixes in deliberately misleading prompts and tests where the agent couldn't run code. Under fair conditions, 86% of patches worked. This is a direct, data-backed challe...