Transformer initialization decides whether sorting generalizes
A two-layer transformer can reach near-zero sorting loss yet fail on longer sequences, so training accuracy alone does not show that it learned a reusable algorithm. The paper finds two solution types selected by weight initialization: one ignores the second attention layer and fails to length-generalize, while the other uses both layers and generalizes perfectly beyond the fixed training length. On finding a sequence minimum, the pattern reverses, showing that extra layers can help or hurt depending on the learned mechanism.