Transformers Learn to Defy Their Own Cluster‑Inducing Bias
A new analysis shows trained Transformers develop representations that push back against the clustering tendency built into self‑attention. The study combines theory from Geshkovski et al. with empirical tests (see the MetastableStateAnalysis repo), revealing that models actively resist architectural collapse—a clue for future model design.