AI refusal is a brittle safety layer, not a guarantee
Modern chatbots are trained to refuse dangerous requests, but that safeguard can fail while the underlying harmful knowledge remains intact. Companies reward refusals, penalize refusals of harmless prompts, and sometimes use AI to train AI on this boundary. The result is a system that may decline questions about weapons or self-harm, yet still retains capabilities that companies say can match top human hackers and sophisticated misinformation operators.