KV cache

How it works

In a transformer, every layer computes a key and a value vector for each token. During decode, each new token attends to all earlier tokens, so serving engines keep those keys and values in GPU memory instead of recomputing them. That store is the KV cache: prefill builds it and each decode step adds one token to it.

Per sequence, its size is roughly 2 (keys and values) x layers x KV heads x head dimension x bytes per value x tokens, so it grows linearly with context length and with the number of sequences being served. Architecture choices such as grouped-query attention or sliding-window attention shrink it, engines can store it at 8-bit, and paged allocation (as in vLLM) reduces wasted memory.

Don't confuse it with prompt caching. The KV cache is the engine's working memory inside inference; prompt caching is a billed provider feature that keeps a prefix's processed state between your requests so later requests skip that prefill. When self-hosting, budget GPU memory for the weights plus the KV cache for your target context length and concurrency.

Worked example

Illustrative: a model with 32 layers and 8 KV heads of dimension 128, stored at 16-bit (2 bytes), needs 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 131 KB, per token. A 32,000-token context needs about 32,000 x 131 KB = 4.2 GB, and eight such requests at once need about 34 GB on top of the weights.

Common mistakes

• 1Sizing a GPU for the weights alone, then running out of memory once long contexts or many concurrent users arrive.
• 2Expecting a billing discount from the KV cache; only the provider's prompt caching changes what you pay.
• 3Configuring the model's full context window on hardware whose KV cache budget only supports a fraction of it.