Netflix powers in‑house LLM inference with Triton and vLLM
Netflix migrated its production LLM serving from TensorRT‑LLM to vLLM, wrapping it in NVIDIA Triton for model loading, batching and GPU scheduling. The new stack lets small models run on‑CPU while large ones use a shared Model Scoring Service, cutting latency and giving engineers full control over rollout and observability.