Agent safety tests rarely record whether systems stop or escape
A new framework shows why agent-safety results are hard to compare: most evaluations do not record whether an agent safely stops, runs out of budget, or escapes its assigned scope. The authors audit 102 evaluations and find 87 with an out-of-scope effect or specification violation, but only 20 that recorded both safe stopping and violation outcomes. Across 22 incidents, the environment allowed an out-of-scope effect in 20, linking agent persistence to permissive boundaries.