Unlocking asynchronicity in continuous batching
HuggingFace explains how to separate CPU and GPU workloads in LLM inference to eliminate idle gaps in continuous batching. By running CPU batch preparation and GPU compute in parallel, asynchronous batching can reclaim nearly a quarter of runtime lost to synchronous scheduling.