How speculative decoding speeds things up, and why batch size decides
What happened
An October 19 Hacker News front-page post explained how speculative decoding works. It does not cut compute per token; total FLOPs for the same sequence go up. The speedup comes from shifting the GPU from memory-bound token-by-token decoding to a compute-bound state, trading idle compute for speed. The gain depends on batch size: when the batch is far below the optimal token count, speculative decoding uses spare compute; at large batches it can waste compute. vLLM offers a --speculative-disable-by-batch-size flag to turn it off by batch size.
Written by AI from the coverage · updated 53 minutes ago
Coverage
Follow the reports to see the story from different sides.
- Hacker News front pageWhy Is Speculative Decoding Fast?
投机解码之所以快，并非因为每个token的计算量更少——相同序列下总FLOPs反而更高，而是因为它把GPU从逐token解码时的内存受限状态转为计算受限状态，用闲置算力换取速度。在batch size远小于最优token数时，投机解码能利用"免费"算力加速；但batch较大时反而浪费算力，因此vLLM提供--speculative-disable-by-batch-size参数可关闭该功能。
Heat over time
Not enough continuous observations to draw a trend yet.