LLM rankings tumble when judged by other LLMs in MUD benchmark
Researchers ran CrucibleBench, a 12‑room MUD where 13 LLMs pursued social goals, and let a Gemini Flash Lite classifier judge their behavior. Rankings shifted up to six places when the judge was ablated, while overall Cohen’s κ was a meaningless 0.04, showing LLM‑as‑judge metrics can hide major failures. The team urges benchmarks to publish per‑subject agreement and judge‑ablation analyses.