Activation Verbalizers Struggle to Reveal Model's Internal Reasoning in Math Tasks
Researchers evaluated activation verbalizers—natural‑language autoencoders that map hidden activations to text—to see if they can surface a model’s chain‑of‑thought while solving math problems. Tests on open‑weight NLAs for Qwen2.5, Gemma, and Llama showed occasional hints of reasoning but overall unreliable reconstruction, limiting their interpretability value.