Skip to content
r/LocalLLaMA

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

RTPurbo converts full-attention LLMs to sparse inference with a few hundred adaptation steps. It keeps the full KV cache only for retrieval heads, uses a 16-dimensional token indexer, and reports up to 9.36x prefill speedup at 1M context plus about 2.01x decode speedup on long-context and reasoning benchmarks.

Why it matters: RTPurbo converts a full-attention model to sparse inference by keeping full KV cache only for retrieval heads, cutting prefill time up to 9.36x at 1M tokens.

Read the original ↗Export Markdown