Hallucination detector rankings invert on small on-device models
Teams deploying 4-bit, on-device Llama-3.2-3B or Qwen2.5-3B models should not copy detector rankings from larger-model studies: the ranking inverted between these two models, making detector choice deployment-specific. In a one-GPU audit on closed-book TriviaQA, a cheap lexical n-gram baseline performed statistically indistinguishably from sampling-heavy methods. The paper also finds a common TruthfulQA labeler nearly useless, with AUROC of 0.46-0.59 versus 0.69-0.89 on TriviaQA.