Tool-using AI judges still flip 40% of correct verdicts
Agent-as-a-Judge systems can overturn up to 40% of their correct verdicts after receiving a single false, authority-attributed note, even when they gather decisive evidence themselves. Across four judges and five datasets, the note changes behavior through over-trust, fabricated support, or stopping after weak corroboration; lightweight mitigations reduce the bias but do not remove it. For anyone using agents to evaluate open-ended work, tool access is not a guarantee of reliable judgment.