Hijacked-agent detectors that score well fall apart at a 2% attack rate
Detectors try to flag hijacked agents from the tools they call, but test sets audited here are 16% to 86% attacks. At a 2% attack rate, a detector with an AUROC, a measure of how well it ranks attacks above normal traces, of 0.879 reaches a precision of 0.076, so most alerts are false. No detector trained on one benchmark, AgentDrift, ranked traces from a second benchmark better than chance, including one that scored 0.996 on that second benchmark itself.