DiLoCo scales predictably, beating traditional data parallel training under fixed compute
DeepMind’s new paper quantifies how the DiLoCo algorithm— which relaxes gradient synchronization across data centers— scales with model size under a fixed compute budget. The authors find that when properly tuned, DiLoCo not only follows a predictable scaling law but also achieves lower loss and larger optimal batch sizes than standard data‑parallel training, even for modest models.