Closed-Form V-Trace Fix Eliminates Bias from Delayed RLHF Rewards
Production RLHF pipelines often receive reward signals minutes after a rollout, breaking PPO’s synchronous‑reward assumption. The paper introduces Retroactive Advantage Correction (RAC), a two‑line patch that injects delayed rewards via a non‑negative kernel, provably removing bias and cutting policy error up to 48× in a tabular MDP. This enables faster, cheaper learning when human or compute‑heavy feedback is lagged.