Tokenizer
The component that splits text into tokens and maps them to numeric IDs from a fixed vocabulary, and back again. Each model family has its own tokenizer.
Why it matters. It decides how many tokens your text counts as, which affects price, context use and speed comparisons across models.
How it works
Most language models use subword tokenizers, such as byte-pair encoding variants, learned from text, with vocabularies from tens of thousands to a few hundred thousand entries. Common words become one token, while rare words, code, numbers and many non-English scripts split into several. Special tokens mark roles, turn boundaries and tool calls in the chat template.
Because tokenizers differ, the same text can be a different number of tokens on different models, and makers can change tokenizers between generations. Providers usually offer a token-counting endpoint or library; counts from another family's tokenizer are only approximate.
When comparing prices, context windows or tokens per second across families, convert to a common unit for your own text, such as cost per document. When self-hosting, use the tokenizer and chat template shipped with the checkpoint.
Worked example
Illustrative: a 10,000-word document is 13,000 tokens on tokenizer A and 15,500 on tokenizer B. At $2 per 1M input tokens on A and $1.80 on B, it costs 13,000 / 1,000,000 x $2 = $0.026 on A and 15,500 / 1,000,000 x $1.80 = $0.0279 on B, so the lower list price is the more expensive one for this text.
Common mistakes
- Comparing per-token prices across model families without accounting for different token counts.
- Estimating tokens with a fixed words-to-tokens ratio for code or non-English text, which often needs many more tokens.
- Using a mismatched tokenizer or chat template when self-hosting, which degrades output without any error.
See it on a real card: Specifications on Llama 4 Maverick.