OLMo3-32B can lose arithmetic generalization, then recover nearby
Pre-training can make OLMo3-32B abruptly switch between solving a task and copying its tempting pattern, so researchers may need to evaluate intermediate checkpoints instead of trusting the final one. On an “answer+1” arithmetic test, its score fell from 81% at 2.17T tokens to 0% at 2.19T, then recovered to 81.7% at 2.21T. The paper calls this mode-hopping and proposes that competing circuits and changing data windows drive it.