RL Updates Concentrate in 5–30% of Model Parameters
Post-training updates touch only 5–30% of parameters across seven RL algorithms and ten models, but this is sparsity, not merely small movement. The post explains the mechanism: on-policy samples make the score’s mean zero, so each parameter’s expected update is its covariance with the reward signal, while the Fisher measures that score’s variance. That variance is small when a parameter moves all candidate tokens together, or when the model is confident.