Chasing Goodput, Not Throughput, Improves LLM Service Experience
A vLLM benchmark on a single NVIDIA A10G showed that a config boosting token throughput by 50 % introduced severe streaming delays, slashing goodput. Goodput—completed requests that meet latency targets like time‑to‑first‑token—better reflects user experience than raw throughput. Tuning for goodput keeps LLM endpoints responsive at scale.