Netflix's in‑house LLM serving stack slashes API costs and boosts control
Netflix chose to run LLM inference inside its existing JVM‑based serving system, using vLLM on NVIDIA Triton with a Java control plane for deployment, autoscaling, and zero‑downtime upgrades. The approach cuts third‑party API spend, enables fine‑grained policy enforcement, and surfaced production trade‑offs like GPU scheduling complexity.