Unlocking Asynchrony in Continuous Batching

解锁连续批处理中的异步性

Hugging Face says an 8B model generating 8K tokens leaves the GPU idle for 24% of the time, and asynchronous batching uses CUDA streams to overlap CPU preparation for batch N+1 with GPU computation for batch N.

Why it matters: HKR-H/K/R all pass, but this is inference-systems engineering rather than a major model release. The Hugging Face post provides a concrete 24% idle-rate number and CUDA-stream overlap mechanism, placing it in low featured.