Best coding setup passes 39.3% of tasks harvested from real vibe-coding sessions
SessionFarmer turns chat-driven coding sessions and their commits into self-contained tasks with a container, task statement, rubrics and unit tests. Its open CropChat benchmark has 242 tasks from 61 public repositories in 14 programming languages. Across 12 model-harness setups the best reaches 39.3% Pass@1, the share solved on the first try, and the harness alone shifts Pass@1 by up to 8.4 percentage points for the same model.