Skip to content
Synced · WeChat

Alibaba RTPurboV2 uses hundreds of training steps for 10x sparse attention

阿里RTPurboV2:原生Transformer再次崛起,百步训练实现10倍稀疏注意

Alibaba’s RTP team released RTPurboV2, replacing 85% of attention heads with SWA and compressing the remaining 15% retrieval heads using low-rank projection, clustering, and dynamic top-p; the adaptation uses about 600 training steps and roughly 1M label tokens, with reported Prefill speedup up to 9.36x.

Why it matters: Alibaba's RTP team swapped 85% of attention heads for sliding window attention, adapting with about 600 training steps and 1 million tokens, for up to 9.36x faster prefill inference.

Read the original ↗Export Markdown