FastDMS: 6.4X KV-cache compression running faster than vLLM BF16/FP8
FastDMS released an MIT implementation that cuts KV memory to 1/5–1/8 of vLLM BF16 at 8K context. A Llama-3.2-1B replication reports PPL 9.200 with 6.4x compression; Qwen3-8B c=1 drops KV from 1.406 GiB to 0.184 GiB. The key detail is physical reclamation of evicted slots, not just nominal KV-byte reduction.
Why it matters: FastDMS open-sourced an MIT-licensed KV cache compression implementation that cuts Llama-3.2-1B's cache to one-fifth to one-eighth at 8K context, with concrete perplexity and memory figures.