PagedAttention loans OS virtual memory to LLM KV caches, doubling GPU serving
The KV cache, not the weights, is an LLM's scarcest resource — and naive serving reserves huge contiguous chunks per request, most of it idle. PagedAttention gives the cache OS-style virtual memory: fixed-size blocks shared across requests and reused as they fluctuate. Fragmentation vanishes, wasted gigabytes shrink roughly 64x, and one GPU serves two to four times more users.