Skip to content
Synced · WeChat

CUHK Pion optimizer updates LLMs on iso-spectral manifolds to address AdamW and Muon instability

AdamW和Muon的失稳根源有解了,港中文新优化器Pion:在等谱流形上更新大模型

CUHK and collaborators introduced Pion, an optimizer that preserves weight singular values through orthogonal equivalence transformations, and reported that it kept a 60M normalization-free LLaMA-like model stable for 9.6B training tokens while AdamW and Muon collapsed with NaNs.

Why it matters: A CUHK team's Pion optimizer keeps singular values fixed and updates only via orthogonal transforms, training a 60M-parameter model for 9.6B tokens where AdamW and Muon hit NaN.

Read the original ↗Export Markdown