Agent‑Safety Benchmarks Mostly Measure Capability, Not Safety
An audit of four popular agent‑safety benchmarks—R‑Judge, InjecAgent, AgentHarm, and AgentDojo—shows their scores track raw capability more than safety. When capability is controlled for, the safety rankings flatten or invert, suggesting the field isn’t actually measuring safety as intended.