Stanford retest finds 44% of LLM research findings no longer hold
What happened
On November 4, The Decoder reported that Stanford researchers Owen Queen and James Zou used 20 models to retest 50 widely cited large language model research findings. They found 44% no longer held and 34% had weakened significantly. Of 22 findings whose expiry could be estimated, the average shelf life was 15.7 months. Findings about which tasks models cannot do expired fastest, while some sycophancy and preference behaviors persisted or grew stronger. Models from the same release window still differed widely. The researchers recommend tying findings to specific models and dates, recording versions and dates, and publishing prompts and analysis code so others can retest.
Written by AI from the coverage · updated 1 hour ago
Coverage
Follow the reports to see the story from different sides.
- The DecoderRapid LLM development is outrunning the research meant to explain it, study finds
斯坦福研究者 Owen Queen 和 James Zou 用20个模型复测50项广泛引用的大语言模型结论，发现44%已失效、34%显著减弱。22项可估算失效时间的结论平均有效期仅15.7个月，关于模型无法完成哪些任务的结论失效最快，部分迎合行为与偏好则持续存在或增强。同一发布窗口内模型差异仍很大，研究者建议将结论限定于具体模型和时间，记录版本与日期，并公开提示词及分析代码以便复测。
Heat over time
Not enough continuous observations to draw a trend yet.