Unlocking Asynchrony in Continuous Batching
解锁连续批处理中的异步性
Hugging Face says an 8B model generating 8K tokens leaves the GPU idle for 24% of the time, and asynchronous batching uses CUDA streams to overlap CPU preparation for batch N+1 with GPU computation for batch N.
Why it matters: Hugging Face shows an 8B model idles its GPU 24% of the time generating 8,000 tokens, and explains how CUDA streams fill those gaps.