What Makes a Benchmark Truly Useful? Five Concrete Criteria
A new arXiv paper distills five precise properties—correctness, solvability, verifiability, clear specification, and meaningful difficulty—that define a good benchmark. It argues that the best tasks mirror real‑world problems practitioners recognize in their own terminology, shifting focus from clever tricks to genuine progress.