BuggyMLE makes ML coding agents face real multi-file failures
BuggyMLE gives researchers a harder, repository-level test for ML coding agents, where the best solve rate falls from 95% to 9% across its easiest and hardest tiers. It injects controlled bugs into working machine-learning repositories, producing more than 6,600 variants from 222 source repos. Training on three difficulty levels also pushed a Qwen3.5-9B model’s SWE-bench Lite resolve rate from 24% to 33%.