Agentic coding gains may belong to the harness, not the model
Swap the harness while holding the model fixed, and the supposed native advantage mostly disappears. Across 792 graded runs, the study found no average winner for either harness, though repository tasks and contest tasks pulled in opposite directions. That makes vendor-native benchmarks look less like model tests and more like bundled-system tests, with cost and timeout handling adding another layer of uncertainty.