Automated AI Scanners Reveal Hidden Flaws in Agent Benchmarks
The paper builds AI scanners that automatically flag four validity flaws—ground‑truth leakage, tool failure, guessing, and answer‑format ambiguity—in agentic benchmark transcripts. It demonstrates the scanners catch real quality issues across five popular benchmarks, exposing hidden weaknesses that manual audits miss.