RG Theory Shows When Transformer Attention Is Relevant or Irrelevant
Using Wilsonian renormalization group theory, the authors treat attention as a perturbation of the MLP residual stack. They find attention is a relevant operator for long‑range data, driving a phase transition and higher effective rank, but irrelevant for short‑range data, where the Transformer behaves like the MLP. The work links attention’s utility to data spectral structure.