One CUDA kernel pushes North Mini Code serving 1.58× past vLLM
At batch size 1, Cohere’s North Mini Code server reaches 292 tokens per second on one H100, versus 185 for vLLM. The trick is a persistent decode megakernel that folds the forward pass into one resident GPU kernel, while still handling continuous batching, paged attention, ragged lengths, and an OpenAI-compatible API. That makes a once-demo-grade optimization look deployable, not merely fast in a benchmark.