Hebbian Rules May Be Sparse Autoencoders in Disguise
A detailed derivation argues that competitive Hebbian updates are not merely analogous to sparse autoencoder training: under tied weights, parts of the update are recognizable SAE descent terms. The omitted encoder gradients, cross-feature interactions, and approximation choices explain why familiar Hebbian rules can learn selective features while still underperforming a properly trained SAE. A small language-model activation experiment tests the connection.