Virtual memory, transplanted onto the LLM KV cache
Naive inference reserves a full max-context KV slot per user, so a short request can hoard thousands of tokens it never uses — both internally and externally fragmented. PagedAttention lifts virtual memory straight from 1960s OS design, paging cache into fixed blocks so kernels still run on non-contiguous memory. The payoff: 2–4× more users served per GPU without bigger hardware.