Prefill and decode

The two phases of generating a response: prefill processes the whole prompt in parallel, and decode then generates output tokens one at a time, each depending on the ones before.

Why it matters. They explain why long prompts delay the first token, long answers slow everything after it, and output tokens cost more than input tokens.

How it works

In prefill, the model processes all prompt tokens in one parallel pass, storing their attention keys and values in the KV cache, and produces the first output token. In decode, it generates one token per step; each step reads the model's weights and the whole KV cache to produce the next token.

Prefill is compute-bound: the GPU does a lot of math per byte read, so it gets through many tokens per second. Decode is usually limited by memory bandwidth, because each step reads all the active weights to produce a single token per request; serving engines therefore batch many requests' decode steps together. This asymmetry is one reason output tokens are priced higher than input tokens.

Time to first token mostly reflects prefill (plus queueing), while output speed reflects decode. Shorter prompts and prompt caching help the first; smaller models, speculative decoding and shorter answers help the second.

Worked example

Illustrative: a server that prefills at 5,000 tokens per second and decodes at 50 tokens per second per request spends 10,000 / 5,000 = 2 s on a 10,000-token prompt and 500 / 50 = 10 s on a 500-token answer. Halving the prompt saves 1 s; halving the answer saves 5 s.

Common mistakes

Trimming the prompt to fix slow responses when most of the time goes into decoding a long answer.
Assuming decode speed is constant; it drops as the context grows because each step reads a larger KV cache.
Forgetting that reasoning tokens are decoded one at a time like visible output, so thinking adds decode time.

See it on a real card: Where to run it on Llama 4 Maverick.