Coding-agent leaderboards hide a 40× cost swing from evaluation harness choice
The authors show that the evaluation harness – the scaffold that supplies tools, enforces context, and decides when to stop – can change token usage per solved task by up to 40×, while model pass‑rate shifts stay under 8 percentage points. This means leaderboard rankings conflate model quality with harness design, so real‑world cost and latency must be reported alongside model specs.