Optimizer rankings can flip when batch size changes
The optimizer that wins language-model pretraining at one batch size may lose at another, so a single-batch benchmark can mislead your training choice. The paper tests whether hyperparameter scaling rules preserve performance as batch size and gradient noise change, and finds no principled scaling rule for Muon that works consistently across training settings. Even after extensive tuning, the best optimizer changes with batch size.