Kimi Linear: A Hybrid Linear Attention That Beats Full Attention
Kimi Linear: An Expressive, Efficient Attention Architecture
Moonshot AI's Kimi team released a tech report on Kimi Linear, a hybrid linear attention architecture. Its core, Kimi Delta Attention (KDA), extends Gated DeltaNet with finer-grained gating to use limited RNN memory more effectively. They trained a 3B-active, 48B-total MoE model mixing KDA and MLA layers. Under the same recipe, it outperforms pure MLA across all benchmarks, cuts KV cache by up to 75%, and boosts 1M-context decoding throughput 6x. The team open-sourced the KDA kernel, vLLM integration, and model checkpoints.
Why it matters: Moonshot AI released Kimi Linear, a hybrid linear attention architecture whose KDA module adds finer-grained gating on top of Gated DeltaNet to use limited RNN memory more efficiently.