Temperature Sampling Misses Model Gaps: Only Diverse Ensembles Reveal True Uncertainty
By sampling a single LLM 100 times at temperature 1 and contrasting it with a 24‑model ensemble at temperature 0, the authors show that temperature‑driven variation lives in at most one dominant dimension, offering per‑question confidence but no cross‑question insight. Only a diverse ensemble surfaces the model’s true unknowns, calling into question the reliance on stochastic sampling for uncertainty estimation.