No backprop needed: FPO adapts LLMs at 3x fine-tuning speed, 40% less memory
Forward-Pass-Only (FPO) training adapts LLMs with no backward pass through the model body, delivering 2.7-3.2x fine-tuning throughput at ~40% less peak memory. It exploits the finding that output-layer prediction error approximates the true gradient at late layers (cosine similarity 0.47-0.59), letting one error signal update each target layer directly. In-domain perplexity improves on OLMo-2-7B, Qwen3-8B and Falcon3-7B while benchmarks stay within seed noise.