Precision formats (BF16, FP8, FP4)

Number formats used to store weights and do the math in training and inference. BF16 uses 16 bits per value, FP8 uses 8 and FP4 uses 4; fewer bits mean less memory and faster math, with less numeric precision.

Why it matters. The precision a model is served at changes its memory use, speed, cost and sometimes its quality.

How it works

BF16 (bfloat16) keeps the wide range of 32-bit floats with fewer precision bits, which made it a standard for training and serving; FP16 has more precision but less range. FP8 comes in variants such as E4M3 and E5M2, named for their exponent and mantissa bits. FP4 formats such as MXFP4 and NVFP4 group values into small blocks that share a scale factor. Integer formats like INT8 and INT4 are also common in quantization.

Hardware support decides what is actually fast: newer GPU generations added native FP8 and later FP4 math, while older ones may store weights at low precision but compute at a higher one. Some models are trained or released natively at FP8 or FP4; others are quantized after training.

Bytes per parameter are 2 for BF16, 1 for FP8 and 0.5 for FP4, plus a little for scales. Hosts may serve the same open model at different precisions, so check what each serves and test lower precisions on your own tasks.

Worked example

Illustrative: a 100B-parameter model needs 100 x 2 = 200 GB at BF16, 100 GB at FP8 and about 50 GB at FP4. With one 8-bit scale per block of 32 values, FP4 storage is (32 x 4 + 8) / 32 = 4.25 bits per parameter, about 53 GB.

Common mistakes

Expecting FP8 or FP4 to run faster on hardware without native support for those formats.
Treating all 4-bit formats as equal; integer, MXFP4 and NVFP4 differ in scaling and in quality loss.
Comparing hosts' prices for the same model without checking which precision each one serves.

See it on a real card: Where to run it on Llama 4 Maverick.