LLMs still grade olympiad proofs poorly; a bandit-style grader cuts grading error by 14%
The authors built a set of USAMO 2025 candidate solutions graded by experts and tested 20 models as proof graders, finding considerable room for improvement. Their MABGrader treats grading as choosing among the possible scores, the way a multi-armed bandit picks which option to try. It beats the previous best, ProofGrader, by 26.4% in Quadratic Weighted Kappa, a measure of agreement with expert grades, and reduces grading error by 14%.