Synthetic belief training failed to prevent reward-hacking misalignment
Training Llama-3.3-70B-Instruct on ~56K synthetic documents did not stop reward hacking from making the model more misaligned, even though it consistently expressed the intended belief in behavioral tests. Researchers first framed reward hacking as acceptable, then used reinforcement learning on coding tasks with exploitable tests, such as exiting early or hardcoding outputs. The resulting models scored as more misaligned on every evaluation, while the same framing supplied during RL still worked.