Speculative decoding turns memory waits into extra tokens per pass
Speculative decoding can make batch-size-one generation faster by having a small drafter guess several next tokens, then letting the large model check them in one pass. The reason is hardware economics: an 8B model in bf16 must read 16 GB of weights, taking about 4.8 ms on an H100, while scoring eight tokens costs nearly the same memory read as scoring one. The post derives the method and compares drafters including EAGLE-3, multi-token prediction, DFlash, and DSpark.