RLHF’s Preference Averaging Silences Minorities, New PA‑RLHF Cuts Fairness Gap
Standard RLHF aggregates heterogeneous human preferences into a single reward model, letting majority groups dominate and creating a procedural fairness failure. The authors propose Preference‑Aware RLHF (PA‑RLHF), which separates reward learning by preference mode, raising alignment accuracy from 46.9% to 67.9% and shrinking the fairness gap from 15.9 to 9.6 points.