Cache‑Based Top‑K Logits Cut LLM Distillation to One GPU
The new paper introduces offline caching of a teacher’s top‑K logits and a fused chunked KL‑loss that avoids the full vocab matrix. These tricks shrink VRAM from ~250 GB to fit on a single GPU, slashing cost and letting anyone run large‑scale LLM distillation.