Chess makes prompt optimization cheap, renewable, and hard to game
Researchers turn 1,118 Lichess puzzles into a benchmark for testing whether better prompts actually improve frozen language models, without relying on static, easily contaminated test sets. Each answer gets cheap exact-match scoring, while engine checks can evaluate other moves; fresh puzzles and adjustable difficulty keep the test renewable. In a study of six optimization methods across eight models, the full run cost about $800, and the strongest evaluated model solved only about 55% of puzzles.