Training Gemma to cite its constitution improves some alignment generalisation
A Gemma 3 27B-IT model trained to reason from a constitution showed better alignment on familiar training scenarios, but the broader generalisation result is still unresolved. The experiment rewards both aligned answers and correct, relevant constitutional citations in the reasoning, then tests whether learning one principle transfers to unseen dilemmas, environments, and principles. Supervised fine-tuning worked best, while reinforcement learning produced only slight evaluation gains and suffered from leaked training data in the held-out-principle test.