DyT lets Transformers Drop Layer Norm without Losing Accuracy
Researchers introduce Dynamic Tanh (DyT), a simple element‑wise tanh scaling that replaces layer‑norm in Transformers. Across vision, language, supervised and self‑supervised tasks, DyT‑enabled models match or surpass the performance of normalized counterparts, challenging the belief that normalization is mandatory.