Matryoshka SAEs Create Hierarchical Features Plain SAEs Miss
Matryoshka Sparse Autoencoders (MatSAEs) train several nested dictionaries together, forcing the latent space to separate general concepts from fine‑grained details. This hierarchy eliminates the feature‑splitting and absorption problems that plain SAEs suffer, making the learned features more interpretable for large language models. The post walks through a guided replication of the original MatSAE paper, confirming the hierarchical effect.