Behavioral tests overstate how much models use their stated reasoning
A new causal audit finds that Qwen3-4B actually uses 76.9% of its stated reasoning steps to produce answers, below the 88.2% score from standard text-editing tests. The method patches a model's internal activations at each reasoning step with those from a counterfactual run, then checks whether the answer changes as predicted. The smaller Qwen3-1.7B reaches 54.8% causal faithfulness, falling from 68% on two-hop tasks to 30% on six-hop tasks.