A small projector cuts Qwen3 prefill time in half
The expensive part of long-context inference may not need every layer’s full KV cache. LLKVApprox uses a small learned projector to approximate later-layer KV values during prefill, then runs the full model for decoding; a Qwen3-8B browser demo reports 341-token prefill completing in half the usual time. The tradeoff is real engineering work in the inference engine, plus careful handling of prompts and the final few thousand tokens.