Aligned fine-tuning can make models unsafe in new contexts
Filtering misaligned examples is not enough: fine-tuning on aligned data can make an LLM behave badly in a different context, so post-training evaluations need to test more than the training examples. The paper calls this “context confusion”: similar representational shifts can make a behavior learned for one domain activate in another, such as carrying data-saving advice from research into a privacy-sensitive app setting. It finds the effect across gender equality, privacy, and physical safety, and reports that targeted alignment data or in-context examples reduce it more than general alignment data.