SGLang introduces Weight Cache Daemon for sub-second engine restarts
SGLang 推出 Weight Cache Daemon,实现亚秒级引擎重启
SGLang released Weight Cache Daemon, a persistent GPU process that holds post-quantized model weights in GPU memory and serves them to new engine instances via CUDA IPC zero-copy mapping. On the Ling-2.6-1T FP8 model, weight loading dropped from ~495s to ~0.63s, a ~785× speedup, and total startup fell from 8.8 minutes to 0.528 minutes. The daemon also enables multi-instance weight sharing on the same GPU, active-standby failover in under 1 second, and multi-node support. This is phase one of SGLang's Fast Engine Recovery Framework, targeting sub-10-second cold restarts for production LLM serving.
Why it matters: SGLang keeps quantized, split weights resident in GPU memory and maps them zero-copy over CUDA IPC when the engine restarts, skipping reload, deserialization and requantization so restarts land under a second.