Time to first token

The time from sending a request until the first output token arrives, measured with streaming on. It covers the network round trip, queueing at the provider and prefill of the prompt.

Why it matters. In chat and voice apps it is the delay users feel before anything happens.

How it works

Time to first token (TTFT) is one part of latency. It includes network time, waiting in the provider's queue and prefill of the whole prompt. For reasoning models, the first token of the visible answer may arrive only after the thinking tokens, although some APIs stream thinking or a summary of it earlier.

TTFT grows with prompt length and with load on the provider. Prompt caching can cut it for long repeated prefixes, because the cached part does not need prefill again. Measured values vary by region, time of day and service tier, so any single figure is a snapshot.

Keep three numbers apart: latency is the time to the complete response, TTFT is the time to the first token, and output speed is how fast tokens arrive after that. For interactive use, a low TTFT with streaming often matters more than total time; for batch jobs it hardly matters.

Worked example

Illustrative: model A has 0.4 s TTFT and 40 tokens per second; model B has 1.5 s and 120 tokens per second. A 20-token reply takes 0.4 + 20 / 40 = 0.9 s on A and 1.5 + 20 / 120 = about 1.67 s on B; a 100-token reply takes 0.4 + 100 / 40 = 2.9 s on A and about 2.33 s on B. Which is faster depends on answer length.

Common mistakes

Measuring TTFT without streaming, which actually measures the time to the full response.
Counting the first thinking token as the start of the answer when users see nothing until the visible answer begins.
Testing with short prompts at quiet times, then shipping long prompts at peak hours.

See it on a real card: Specifications on Llama 4 Maverick.