Data, not model tricks, drove most pretraining gains
The usual story of AI progress puts the spotlight on architectures and optimizers. A 2019-to-2025 comparison finds data improvements delivered 12x compute efficiency, versus 3.7x from model improvements, with the gains largely additive rather than entangled. The result is based on small-scale pretraining experiments and OLMES capability tests, but it still shifts attention toward corpus construction, filtering, and data quality as frontier advantages.