Train an LLM only on fifth-grade text and it still caps out: pretraining data is the ceiling
LittleLearner trains 0.6B–5B models from scratch on an 88B-token corpus filtered to U.S. elementary-school content, with matched unfiltered controls. Scaling, RL, and in-context learning amplify in-scope skills, but none push capability past the K–5 curriculum — suggesting pretraining data sets the true capability ceiling.