Skip to content
Synced · WeChat

AMD paper: FP4 training instability is not caused by insufficient randomness

AMD新论文颠覆认知:FP4训练不稳定,原因不是随机性不足

AMD and Penn State pretrained Llama 3.1-8B with MXFP4 on MI355X native FP4 hardware, achieving 9-10% end-to-end speedup over an FP8 baseline, while the paper identifies Wgrad quantization as the bottleneck that raises token overhead to 26-27% without deterministic Hadamard stabilization.

Why it matters: AMD and Penn State pretrained Llama 3.1-8B on MI355X with native FP4, running 9-10% faster end to end than an FP8 baseline, while weight gradient quantization added 26-27% cost per token.

Read the original ↗Export Markdown