Skip to content
Hacker News front page

Do Transformers Need Three Projections? Systematic Study of QKV Variants

Do transformers need three projections? Systematic study of QKV variants

Ali Kayyam and coauthors evaluate three QKV projection-sharing variants across synthetic, vision, and language-modeling settings, including 300M and 1.2B parameter models trained on 10B tokens; Q-K=V halves the KV cache with a 3.1% perplexity degradation, while Q-K=V plus MQA reduces cache use by 96.9%.

Why it matters: An ICML 2026 paper tests three parameter-saving QKV sharing schemes and adds 2D positional encoding to break the attention-map symmetry in two of them.

Read the original ↗Export Markdown