An Algorithm Finds Near‑Perfect Tokenizers—Why It May Not Boost LLMs
A new integer‑linear‑programming technique can compute an optimal tokenizer for a fixed vocabulary size, using cutting‑plane methods akin to solving a TSP. In practice the gain over byte‑pair encoding is under 1% and may not generalize, so the efficiency lift is marginal.