A calibrated judge raises MATH500 accuracy by 4.65 points
A reference-free reward model lets RL train a 1.7B language model nearly as well as an answer key, raising MATH500 accuracy from 72.2% to 76.8%. It works by asking an LLM a typed question about each response, such as whether the final answer is correct, then using its calibrated probability rather than a raw score. The reward's calibration error was 0.074, versus 0.141 for a softmax judge, but it became overconfident on problems the base model mostly failed.