Induction Heads Perform N‑Gram Interpolation, Linking Transformers to Classical Smoothing
The paper shows that induction heads in transformers trained on order‑k Markov chains implement soft context‑matching and Dirichlet‑style smoothing, effectively interpolating n‑grams like Jelinek‑Mercer. This mechanistic insight explains how transformers regularize in‑context learning and can match or beat classical count‑based models.