UpLoop trains looped Transformers at no extra compute, lifting a 1.3B model's average to 27.52
UpLoop starts with a shallow Transformer, grows its depth by copying learned layers, and spends the compute saved early on re-running a shared middle block later in training. It matches a standard dense baseline in final parameter count, training tokens and training FLOPs. At 1.3B parameters, the average score across 22 downstream tasks rose from 22.99 to 27.52, and gains carried over to math reasoning after fine-tuning.