Skip to content
Computing Life · Share · Yage

Kimi K3 tech report: scaling as a set of constrained production factors, not a single knob

Kimi K3 报告解读:当规模从一个旋钮变成一组受约束的生产要素

Moonshot AI released the Kimi K3 tech report: 2.78T total params, 104.2B active per token, 93 layers, native 1M context. The core thread isn't parameter count—it's how the team navigated four hardware walls: VRAM, bandwidth, communication, and latency. On the sequence axis, 69 KDA layers propagate history at constant cost while 24 Gated MLA layers do global correction at a 3:1 ratio, keeping KV cache in check. For depth, Block AttnRes groups 93 layers into 9 block-level addressing sources, slashing cross-device activation transfers. The MoE layer uses LatentMoE to halve communication payloads, with Quantile Balancing and MoonEP smoothing out load skew. Training signals come from AgentENV sandboxes with physical verifiers and dynamic harness swapping—no reward for smooth-talking the judge. Post-training splits domain × inference effort into a 2D matrix of 9 teachers, then distills them into one model via MOPD. Deployment uses QAT throughout: MXFP4 for routed expert weights, MXFP8 for activations, paying the quantization cost during training. The report's real value isn't a single breakthrough—it's a worked example of solving scaling laws under real hardware constraints.

Why it matters: Moonshot AI's Kimi K3 technical report is read for its engineering trade-offs on sequence architecture under memory, bandwidth, communication and latency limits, not for the 2.78 trillion parameters.

Read the original ↗Export Markdown