Output speed (tokens per second)

How it works

Output speed measures the decode phase: tokens generated per second for one request after the first token. It is different from throughput, the total tokens per second a server produces across all the requests it is batching.

Speed depends on model size (active parameters for mixture-of-experts models), hardware, serving engine, batch size, context length and techniques such as speculative decoding. Providers trade per-request speed against throughput: bigger batches serve more users per GPU but make each stream slower, and some sell faster tiers or modes at a higher price. Tokenizers differ, so tokens per second is not directly comparable across model families.

Reasoning models generate thinking tokens at the same speed even when users don't see them, so the visible answer arrives later than the raw speed suggests. Measure with your own prompts and typical output lengths.

Worked example

Illustrative: a 2,000-token answer at 50 tokens per second takes 2,000 / 50 = 40 s to generate; at 150 tokens per second it takes about 13.3 s. If the model first thinks for 3,000 tokens, the 50 tokens-per-second model needs (3,000 + 2,000) / 50 = 100 s.

Common mistakes

• 1Comparing tokens per second across model families with different tokenizers, where the same text is a different number of tokens.
• 2Confusing per-request output speed with server throughput when sizing a self-hosted deployment.
• 3Ignoring hidden reasoning tokens, which take generation time even when they are not shown.