Even Near‑Perfect AI Alignment Can Collapse Human Values
New theory shows that if an AI optimizes based on an imperfect proxy, its value function can dip below any safety threshold, even when the proxy is almost accurate. This explains why over‑optimizing AI poses a catastrophic risk and argues for designs that cap optimization pressure rather than rely on approximate alignment.