Grouped Query Experts Cut Transformer Attention Costs in Half
The paper introduces Grouped Query Experts (GQE), a Mixture‑of‑Experts layer built on grouped‑query attention that routes each token through only a few query‑head experts while keeping key‑value heads dense. On a 250M‑parameter model with a fixed 30B‑token budget, GQE halves query‑head computation yet matches baseline accuracy, enabling cheaper long‑context inference.