Topology-aware scheduling cuts KV‑cache transfer latency 3‑18× in disaggregated LLM inference
When prefill and decode run on separate GPU pools, moving the 2.6 GB KV cache can dominate latency. The authors build a topology‑aware orchestrator that picks the fastest interconnect (NVLink, InfiniBand, or TCP) and pipelines transfers, shaving 60‑85 % of wait time and achieving 3‑18× lower latency in analytical models. This makes large‑scale disaggregated inference viable.