A llama.cpp CPU patch cuts Q5_K prompt time by up to 4x
A proposed llama.cpp CPU kernel makes prompt processing much faster for quantized models, with a reported 4x gain on VNNI hardware and 1.6x on AVX2 in a Q5_K test. It replaces the usual vector dot-product path with tiled matrix multiplication, using 8 threads on a 27B Qwen model and a 2,048-token prompt; the reported gains vary by quantization type, reaching 2.09x in the AVX2 benchmark table.