Skip to content
Trending storyPast story

Weight-editing experiment shows risk of lasting fact tampering

1 report1 sourceupdated 3 days ago

What happened

AI digest

On November 14, Hacker News' front page covered a controlled experiment in model weight editing. Researchers used ChatGPT 6.1-Sol as the editor to modify the weights of Qwen2.5-1.5B-Instruct so that it answered "Buenos Aires" for the host city of the 2000 Olympics on all 100 held-out samples. The authors call this persistent behavior shift from unauthorized editing "weight-space misfolding" and observed side effects including over-refusal and selective omission. The paper argues for governing weight editing with edit permissions, patch records and rollback mechanisms rather than banning its legitimate uses.

Written by AI from the coverage · updated 3 days ago

Coverage

Follow the reports to see the story from different sides.

Nov 14
  1. Hacker News front page
    Weight Space Misfolding

    一项受控实验用 ChatGPT 6.1-Sol 作为编辑器修改 Qwen2.5-1.5B-Instruct 的权重,使模型在 100 个留出样本中全部将 2000 年奥运会主办城市错答为“布宜诺斯艾利斯”。作者将这种越权编辑导致的持久行为偏移称为“权重空间错折”,并观察到过度拒答和选择性遗漏等附带影响。论文主张通过编辑权限控制、补丁记录与恢复机制来治理权重编辑,而非禁止正当用途。

Heat over time

Heat now 3·Comparable peak 9(Nov 14 05:00)·Comparable change over 24 hours -51%

02.557.510Nov 1405:00Nov 1420:00Nov 1511:00Nov 1602:00

The trend only compares accounts observed without gaps, so its range may be smaller than the current heat. Hover or tap the chart for each hour; the left and right arrow keys step through it.