Tokenizers v1 makes text encoding tens of times faster
Hugging Face’s tokenizers v1 is built to stop the CPU tokenizer from leaving GPUs idle, with encoding often tens of times faster than v0.23. It keeps the same token IDs, API, vocabulary, merge ranks, and model-family coverage, so existing workflows can preserve their outputs while processing larger workloads. The project publishes tokbench benchmarks and a command to rerun them on your own hardware.