Ten LLM judges can carry only 3.5 judges’ worth of evidence
Agreement from ten LLM judges may be far weaker evidence than it looks: their errors are correlated, so benchmark and model comparisons can appear statistically decisive when they are not. Across the paper’s ten-judge bank, average pairwise error correlation was 0.21, leaving roughly 3.5 independent judges’ worth of information; ignoring shared errors changed the conclusion of significance in up to 28% of comparisons. The authors recommend trusted examples to measure judge accuracy and shared mistakes before choosing a voting method.