XYBench tests whether AI can solve the real problem
XYBench gives researchers 8,115 questions designed around misconceptions, so they can test whether an AI assistant fixes the user’s underlying problem instead of blindly answering the request. In one example, a user asks how to make frosting stick when the cupcakes are too warm; the right response addresses the heat first. Five evaluated models often stayed focused on the mistaken approach, unlike human reference answers.