Token

How it works

A token is a chunk of text produced by a model's tokenizer: often a whole common word, a piece of a longer word, a punctuation mark or a space. Everything a model reads and writes is converted to tokens first, and prices and limits are counted in them.

Each model family has its own tokenizer, so the same text can become a different number of tokens on different models. A rough rule for English prose is about 0.75 words per token, but code, numbers, unusual formatting and many non-English languages produce more tokens per word. Images, audio and PDFs are converted into tokens too, by rules each maker documents.

Because prices, context windows and output limits are all in tokens, per-token prices can mislead when tokenizers differ. Most APIs return token counts with each response or offer a counting endpoint; use those instead of estimates.

Worked example

With the rule of about 0.75 words per token, a 1,500-word English document is roughly 2,000 tokens. If another model's tokenizer produces 15% more tokens for the same text, it counts about 2,300, and at the same per-token price it costs 15% more.

Common mistakes

• 1Estimating tokens from word or character counts for code or non-English text, where the ratio is often much worse than for English prose.
• 2Comparing per-token prices across makers without checking how many tokens each tokenizer produces for your actual text.
• 3Forgetting that images, PDFs, tool definitions and the system prompt all consume input tokens on every request.