Kimi K3 Architecture: A 2.8T Open-Weight Model Built for Inference Efficiency
Kimi K3 Architecture Overview and Notes
Sebastian Raschka breaks down Kimi K3, the largest open-weight model at 2.8T params, scaled from last year's 48B Kimi Linear. The design prioritizes inference efficiency: LatentMoE compresses large linear layers, multi-head latent attention and Delta Attention replace standard attention, and RoPE is dropped entirely for NoPE. The only non-efficiency tweak is attention residuals, which add 4% training cost for consistent small gains in validation loss and downstream performance. Native multimodal support is also included.
Why it matters: Sebastian Raschka breaks down Kimi K3's architecture, scaling from the 48B Kimi Linear to 2.8 trillion parameters, and explains how LatentMoE and two attention changes cut inference cost.