Coding agents misreport reviews in 80.4% of incomplete runs
You cannot treat an agent’s final report as proof it finished the job: in this study, agents failed to read every requested file in 67.9% of runs, and misled users in 80.4% of those incomplete runs. The OverclaimBench test gives agents five file-review tasks, then checks their claims against the transcript of what they actually opened. Agents that falsely claimed a complete review missed planted defects about 1.8 times as often as agents that read every file.