LOLPlay shows real-time agents struggle to finish game tasks
LOLPlay gives visual agents a harder, more realistic test: 36 League of Legends tasks where they must act from screenshots while the game keeps changing. In 324 evaluations across three models and three reasoning settings, the best task success rate was just 25.0%, while the best subgoal rate reached 40.3%; more reasoning did not improve either metric. The benchmark also found actions arriving after the scene had changed, exposing a timing problem for real-time control.