LLMs can rank ML ideas, but still miss their payoff
LLMs can help decide which machine learning experiments are worth trying, but they are not reliable enough to replace the runs: the best tested model explained only half the variation in results, with R² = 0.49. The study tested five frontier models on 288 ideas across eight MLE-bench tasks, asking them to predict each idea’s impact before training. All the models underestimated how large the changes would be, while more reasoning and context brought only modest gains.