Standard softmax code has two rounding bugs that cause training loss spikes, paper finds
Standard softmax and cross-entropy code, the functions that turn a model's scores into probabilities and a training loss, have two rounding bugs, the authors show. Softmax Collapse zeroes the gradient of a very confident correct answer, so gradients no longer sum to zero; Softmax Cancellation leaves non-zero gradients when a student model already matches its teacher. Their drop-in rewrite removes the Slingshot loss spikes and stops students drifting from teachers in distillation.