Retries can keep an overloaded LLM engine from recovering
For teams serving LLMs through clients or proxies, cancelling timed-out work is not enough: with retries enabled, vLLM restored service in none of the overload runs, delivering only 12% to 38% of calls that arrived afterward. The engine often began retry work less than half a second before timeout, then computed 79% to 85% of the prompt before aborting it. A retry budget or request coalescing worked better, though cancellation could still lose calls.