Claude's quirky tokenizer packs just ~15,000 tokens. The why is the interesting part
A reproduction of Claude's tokenizer suggests Anthropic has been shrinking the vocab — roughly 50,000 entries in Claude 3 down to ~16,000 today — while rivals push into the hundreds of thousands. One plausible motive: the softmax head is a gradient bottleneck that compresses the backprop signal to rank 2D, so a huge vocab wastes information. It may also be why Claude sidesteps glitch tokens and big memory-hungry projection layers.