Debate beats single-judge RLAIF: 45% less reward hacking, holds through RL steps
Dropping single-LLM-judge reward for two-agent debate recovers 45% of the accuracy gap lost to reward hacking in RLAIF, and holds through many RL steps where the baseline degrades. Tested on a Gemini 2.5 Flash policy against a weaker judge; capping the critic's words is what keeps the game from drifting into own judge-hacking.