Program Distillation Turns LLM Judges into Transparent, Low‑Cost Evaluators
The authors distill the decision logic of LLM‑as‑a‑judge into a committee of executable programs, creating a fast, cheap, and inspectable evaluation pipeline. Their PAJAMA system matches a 13B LLM judge across multiple benchmarks while slashing per‑sample API costs, and even yields superior reward models on RewardBench.