vLLM’s push mode cuts disaggregated inference latency under load
vLLM users running prefill and decode on separate instances can now move KV-transfer setup off the time-to-first-token path, making responses arrive sooner under heavy traffic. The push-based NIXL connector coordinates the transfer while decoding is getting ready instead of waiting for the decode instance to pull each block. In a Qwen3-32B benchmark on AWS P5en instances with EFA, it delivered up to 3x better TTFT and 30% better time per output token than pull mode, depending on workload and configuration.