Attention‑Only Transformers Nearly Match Standard Models by Adding Depth
A controlled pre‑training study shows that stripping feed‑forward layers from decoder transformers (Simple Attention Networks) and reallocating the freed parameters to deeper attention stacks cuts the loss gap to just 0.006 nats, even at 105 B‑token scales. The result challenges the assumed necessity of FFNs and points to attention depth as the primary lever for efficient scaling.