LodeHQ
AI news for people who shipFree, by email

AI · arxiv.org · Monday, 9 November 2026

Retries can keep an overloaded LLM engine from recovering

For teams serving LLMs through clients or proxies, cancelling timed-out work is not enough: with retries enabled, vLLM restored service in none of the overload runs, delivering only 12% to 38% of calls that arrived afterward. The engine often began retry work less than half a second before timeout, then computed 79% to 85% of the prompt before aborting it. A retry budget or request coalescing worked better, though cancellation could still lose calls.

Read the source: arxiv.org ↗In the 9 November issue →