Exact dynamics reveal why depth and init boost linear concepts
The paper derives exact solutions for how linear concept directions form during training, showing that data and target geometry, depth, and initialization scale dictate the final abstraction. It extends the theory to nonlinear nets, revealing that ReLU and erf activations attenuate abstraction differently, and validates the findings on models like DINOv3 and Gemma 4, improving linear probe generalization in LLMs.