Ternarizing Qwen3-4B to 1.64 bits: storage halves, but inference gets 4.6x slower
A post-training pipeline squeezed instruction-tuned Qwen3-4B into ternary weights at 1.641 effective bits per weight, shrinking the stored model from 8.29 GiB to 3.96 GiB with perplexity essentially unchanged. But the "1.58-bit" label flatters the result: scored accuracy falls from 64.5% to 54.7%, and degradation is strikingly uneven, from BoolQ keeping 84.6% of teacher performance to ARC-Challenge keeping just 43.8%. And compression doesn't buy speed: a preliminary Triton GEMV kernel ran 4.6x slower than FP16 cuBLAS.