Continuous batching

A serving technique that adds and removes requests from the running batch at every decode step, instead of waiting for a whole batch to finish. It keeps the GPU busy when requests have different lengths.

Why it matters. It is a main reason modern serving engines get far more throughput from the same hardware than one-request-at-a-time serving.

How it works

Because decode is limited by memory bandwidth, a GPU can produce a token for many sequences in not much more time than for one, so engines batch requests. With static batching, a batch runs until its longest request finishes and finished slots sit idle. Continuous batching (also called in-flight or iteration-level batching) schedules at every step: finished sequences leave and waiting ones join straight away.

Serving engines such as vLLM, SGLang and TensorRT-LLM implement it, usually together with paged KV-cache memory and chunked prefill, which splits long prompts so they don't stall other requests' decode steps. The KV cache memory available sets how many sequences fit in the batch.

The trade-off is throughput against per-request speed: bigger batches give more total tokens per second and a lower cost per token, but each stream decodes more slowly and new requests can wait longer for their first token under load. This is why API speeds vary through the day, and why self-hosted cost per token depends so much on concurrency.

Worked example

Illustrative: a GPU decodes one stream at 60 tokens per second, or 32 streams at 40 tokens per second each, 32 x 40 = 1,280 tokens per second in total. At $2 per GPU-hour that is 1,280 x 3,600 = 4,608,000 tokens an hour, about $0.43 per 1M output tokens, versus 216,000 tokens an hour and about $9.26 per 1M for a single stream.

Common mistakes

Estimating self-hosting cost from single-request speed, which overstates the cost per token many times over.
Benchmarking a server with one request at a time and expecting the same per-user speed under production load.
Setting context and batch limits so high that the KV cache runs out and the engine preempts or rejects requests.