Tensor and pipeline parallelism

Ways to split one model across several GPUs. Tensor parallelism splits each layer's matrices across GPUs; pipeline parallelism gives each GPU a consecutive block of layers.

Why it matters. They are needed when a model's weights and KV cache don't fit on one GPU, and the choice affects speed and the interconnect you need.

How it works

With tensor parallelism, every layer is split, so all GPUs work on every token and exchange partial results within each layer. With pipeline parallelism, the first GPU runs the first block of layers and passes activations to the next, so each token moves through the GPUs in turn. Mixture-of-experts models can also use expert parallelism, placing different experts on different GPUs.

Tensor parallelism communicates constantly, so it needs a fast interconnect such as NVLink inside one server, and in return it lowers per-token latency. Pipeline parallelism communicates less and suits splitting across servers, but adds per-token delay and needs many requests in flight to keep every stage busy. A common setup is tensor parallelism within a server and pipeline parallelism across servers; engines such as vLLM, SGLang and TensorRT-LLM expose both as settings.

Parallelism buys memory, not free speed: communication overhead means two GPUs rarely give twice the throughput. Use the fewest GPUs that fit the weights plus KV cache, then add replicas to serve more traffic.

Worked example

Illustrative: a model with 140 GB of weights runs with tensor parallelism across four 80 GB GPUs, 35 GB of weights each, leaving 320 - 140 = 180 GB in total (45 GB per GPU) for KV cache and buffers. With 8 GPUs, two such 4-GPU replicas often serve more users than one 8-GPU group, because each has less communication overhead.

Common mistakes

Running tensor parallelism across servers over a slow network, where communication dominates.
Using pipeline parallelism at low concurrency, where most GPUs wait idle for their stage.
Picking a tensor-parallel size that doesn't divide the model's attention heads evenly, which engines typically reject.

See it on a real card: Where to run it on Llama 4 Maverick.