A True 2-bit KV Quantization Algorithm for Long-context Reasoning Beyond TurboQuant
超越TurboQuant,面向长上下文推理的真2-bit KV Quantization算法问世
TogetherAI and collaborators released OSCAR, a 2.28 BPE INT2 KV Cache system integrated with SGLang, reporting up to 3× decode speedup at 100k context and up to 7× job-level throughput under a fixed memory budget.
Why it matters: TogetherAI released OSCAR, which compresses the KV cache to 2.28 bits per value on average and plugs into SGLang, decoding up to 3x faster at 100K tokens and up to 7x throughput at fixed memory.