Do Transformers Need Three Projections? Systematic Study of QKV Variants
Do transformers need three projections? Systematic study of QKV variants
Ali Kayyam and coauthors evaluate three QKV projection-sharing variants across synthetic, vision, and language-modeling settings, including 300M and 1.2B parameter models trained on 10B tokens; Q-K=V halves the KV cache with a 3.1% perplexity degradation, while Q-K=V plus MQA reduces cache use by 96.9%.
Why it matters: An ICML 2026 paper tests three parameter-saving QKV sharing schemes and adds 2D positional encoding to break the attention-map symmetry in two of them.