Latency

How it works

Latency is the time between sending a request and getting the result. Two numbers matter: time to first token (how quickly streamed output starts) and total time (until the last token). Output speed, in tokens per second, connects the two.

Longer prompts raise time to first token, longer outputs raise total time, and reasoning adds time before the visible answer starts. Makers rarely publish exact figures; some give relative labels, and real numbers depend on load, region, service tier and settings such as effort or fast modes.

Measure latency with your own prompts at your real traffic times, with streaming on if users watch the output. Smaller models, lower effort, prompt caching and priority or fast options are the usual levers.

Worked example

Illustrative: a model with 0.8 s to first token that generates 50 output tokens per second takes about 0.8 + 600 / 50 = 12.8 s for a 600-token answer. Halving the answer to 300 tokens brings it to about 6.8 s.

Common mistakes

• 1Benchmarking with tiny prompts, then shipping prompts many times longer.
• 2Not streaming in user-facing apps, so people wait for the full reply instead of the first token.
• 3Ignoring reasoning time: a model can look fast on short answers and slow once it starts thinking.