Skip to content
AI HOT (Curated Pool)

SGLang adds Waterfill and LPLB to improve DeepEP MoE load balancing

SGLang 引入 Waterfill 与 LPLB 提升 DeepEP MoE 负载均衡

LMSYS and NVIDIA introduced two dispatch-time load balancers in SGLang for DeepEP MoE inference. Waterfill handles the shared expert by sending it to the least-loaded GPU; it boosts throughput by 1.48–4.66% on DeepSeek V3/R1 and lifts DeepSeek V4 from 49,253 tok/s to 51,677 tok/s (+4.92%). LPLB uses linear programming to route tokens across redundant expert replicas, adding 0.84–7.34% throughput on the same benchmarks. Both methods preserve model accuracy. The post reports throughput gains but does not disclose latency impact.

Why it matters: LMSYS and NVIDIA added two runtime load-balancing methods, Waterfill and LPLB, to DeepEP's MoE inference in SGLang, with throughput gains reported on V3, R1 and V4.

Read the original ↗Export Markdown