Low-Precision Training Can Freeze Weights—Predictable Failure Mode Identified
When a weight update falls below half a ULP, that coordinate stops moving even though gradients remain nonzero, causing training to silently stall. The authors derive a simple half‑ULP condition that predicts the freeze from a full‑precision run, and show stochastic rounding eliminates it. This gives a cheap way to guard low‑precision models.