Kimi K3's 2.8T open weights: day-zero on 8x B300, ~460 tok/s
Moonshot's Kimi K3 — 2.8T-parameter MoE, just 16 of 896 experts active per token — isn't easy to serve: it wants an 8x B300 footing. What makes a 3T-class open model hostable at all is MXFP4 weights quantized from SFT onward and a KDA rework baked into vLLM. A user's plain vLLM run on 8x B300 lands near 92 tok/s decode at ~$190 per million tokens; Modal's custom speculator pushes interactive serving to 460 tok/s.