Transformer FFNs Vary Widely in Linearity, Learning Determines It, Not Architecture
The authors measure each feed‑forward block’s linear recoverability (R²_lin) via a closed‑form least‑squares fit and find a stark, non‑monotonic spread—from near‑linear (>0.99) to highly nonlinear (<0.3)—across blocks in GPT‑2, Pythia‑160M and LLaMA‑160M. This variability isn’t set by the activation function; it’s a property learned during training, opening new avenues for targeted compression and analysis.