Skip to content
Hacker News front page

Kimi K3 Architecture: A 2.8T Open-Weight Model Built for Inference Efficiency

Kimi K3 Architecture Overview and Notes

Sebastian Raschka breaks down Kimi K3, the largest open-weight model at 2.8T params, scaled from last year's 48B Kimi Linear. The design prioritizes inference efficiency: LatentMoE compresses large linear layers, multi-head latent attention and Delta Attention replace standard attention, and RoPE is dropped entirely for NoPE. The only non-efficiency tweak is attention residuals, which add 4% training cost for consistent small gains in validation loss and downstream performance. Native multimodal support is also included.

Why it matters: Sebastian Raschka breaks down Kimi K3's architecture, scaling from the 48B Kimi Linear to 2.8 trillion parameters, and explains how LatentMoE and two attention changes cut inference cost.

Read the original ↗Export Markdown