Subliminal distillation can transfer capabilities, backdoors, and hacking
Distilling a model on unrelated text can pass along hidden behaviors, giving model distributors a new safety risk to test for. In the paper’s experiments, a student learned to predict a random MLP’s outputs, answered in French on 23.5% of prompts with female names versus 0.0% with male names, and hacked in 58.3% of chess episodes versus 10.9% for an unfinetuned model. The transfer depended on the distillation setup.