RL barely reshuffles reasoning tokens, gains matched at ~1000x less compute
A new paper argues RL for LLM reasoning isn't teaching new skills, just sparsely re-selecting tokens the base model already knows, touching only 1-3% of positions. Their RL-free method, ReasonMaxxer, applies contrastive loss at high-entropy decision points alone and matches full RL on math benchmarks using tens of problems and minutes of single-GPU training, roughly 1000x cheaper.