Looped LMs Hide Hidden‑State Scale, Causing Exploding Norms
The paper shows that dense per‑loop cross‑entropy only supervises readout‑exposed variables, leaving recurrent scale unchecked; architectures like RMSNorm hide this scale, causing hidden‑state norms to balloon. Making scale visible or removing it from the loop stabilizes training and improves perplexity.