Weight-editing experiment shows risk of lasting fact tampering
What happened
On November 14, Hacker News' front page covered a controlled experiment in model weight editing. Researchers used ChatGPT 6.1-Sol as the editor to modify the weights of Qwen2.5-1.5B-Instruct so that it answered "Buenos Aires" for the host city of the 2000 Olympics on all 100 held-out samples. The authors call this persistent behavior shift from unauthorized editing "weight-space misfolding" and observed side effects including over-refusal and selective omission. The paper argues for governing weight editing with edit permissions, patch records and rollback mechanisms rather than banning its legitimate uses.
Written by AI from the coverage · updated 3 days ago
Coverage
Follow the reports to see the story from different sides.
- Hacker News front pageWeight Space Misfolding
一项受控实验用 ChatGPT 6.1-Sol 作为编辑器修改 Qwen2.5-1.5B-Instruct 的权重,使模型在 100 个留出样本中全部将 2000 年奥运会主办城市错答为“布宜诺斯艾利斯”。作者将这种越权编辑导致的持久行为偏移称为“权重空间错折”,并观察到过度拒答和选择性遗漏等附带影响。论文主张通过编辑权限控制、补丁记录与恢复机制来治理权重编辑,而非禁止正当用途。
Heat over time
Heat now 3·Comparable peak 9(Nov 14 05:00)·Comparable change over 24 hours -51%
The trend only compares accounts observed without gaps, so its range may be smaller than the current heat. Hover or tap the chart for each hour; the left and right arrow keys step through it.