GPU memory (VRAM)

How it works

Weights take about parameters x bytes per parameter: 2 bytes at BF16 or FP16, 1 at FP8 or INT8, and about 0.5 at 4-bit, plus a little for scale factors. On top come the KV cache, which grows with context length, layers and batch size, and activation buffers and engine overhead. For mixture-of-experts models every expert must be loaded, so size by total parameters.

Bandwidth matters as much as capacity: each decode step reads the active weights, so memory bandwidth caps per-request output speed. When a model doesn't fit on one GPU, it is split across several with tensor or pipeline parallelism. Unified-memory machines, such as Apple silicon, give the GPU a large shared pool, usually with less bandwidth than data-center GPUs.

Plan for weights, plus KV cache for your target context and concurrency, plus 10-20% headroom. Engines often reserve a fixed share of memory for the KV cache up front, so a model that loads may still not serve your workload.

Worked example

Illustrative: a 70B-parameter dense model needs 70 x 2 = 140 GB for weights at BF16, 70 GB at FP8 and about 35 GB at 4-bit. On two 80 GB GPUs (160 GB), BF16 leaves about 20 GB for KV cache and buffers, while FP8 leaves about 90 GB, room for far longer contexts or more users.

Common mistakes

• 1Sizing for the weights only and forgetting the KV cache, activations and engine overhead.
• 2Sizing a mixture-of-experts model by active parameters instead of total parameters.
• 3Treating a unified-memory machine as equal to a data-center GPU of the same capacity, ignoring the bandwidth gap that limits speed.