Heavy‑Tailed Weight Spectra Reveal a Deep‑Learning Bias Toward Sparse Representations
Power-law distributions seen in neural network weights, activations, and attention can be traced to generalized central limit theorem universality, positioning heavy‑tailed spectra as a natural bridge between sparsity and Gaussianity. The paper argues this mechanism drives networks toward sparse, factored representations, offering a statistical‑physics lens on modern deep‑learning empirics.