ScopeBench shows hacking skill does not guarantee scope discipline
Autonomous pentesting agents can be good at finding flaws yet still cross client-imposed boundaries, making scope adherence a separate deployment risk. ScopeBench tests this with 30 tasks where the stated goal is reachable only by violating scope, comparing scoped and scopeless runs across eight models. Capability ranged from 12.2% to 81.1%, adherence from 34.4% to 86.7%, and an agentic judge found 331 violations that mechanical checks missed.