Skip to content
r/LocalLLaMA

Shard - Getting to 10× KV Cache Compression

Shard - getting to 10× KV cache compression

Shard reduces Llama-3.1-8B KV memory by about 10× at 8K context and 11× at 32K, with no measured drop on NIAH or LongBench, using PCA plus int4 quantization for K and Hadamard rotation plus vector quantization for V.

Why it matters: Shard compresses KV cache memory by about 90% on Llama-3.1-8B, roughly 10x at 8K context and 11x at 32K, using PCA plus int4 for keys and Hadamard rotation plus vector quantization for values.

Read the original ↗Export Markdown