aiaimodel.cards
Weights & training47

Quantization

Lower-precision weights to cut memory and cost

Quantization

Storing and running a model's weights at lower numeric precision, such as 8-bit or 4-bit instead of 16-bit, to cut memory use and cost.

Why it matters. Lets larger models fit on smaller hardware, at some risk to quality that you need to test.

How it works

Quantization converts a model's weights, and sometimes its activations or cache, from a higher-precision number format to a lower one, for example from 16-bit to 8-bit or 4-bit. The model takes less memory and can run faster or on cheaper hardware.

There are many methods and formats, and the quality loss depends on the method, the bit width and the model. 8-bit is often close to the original, while lower widths can degrade more, especially on reasoning, math and code. Some makers publish official quantized versions, and hosts may serve quantized versions, which they should state.

When comparing hosted prices for an open model, check the precision each host serves. Evaluate a quantized model on your own tasks instead of assuming it matches the original's benchmark scores.

Worked example

Illustrative: a 30B-parameter model needs about 60 GB for its weights at 16-bit (2 bytes per parameter), 30 GB at 8-bit and 15 GB at 4-bit. The 4-bit version fits on a single 24 GB GPU with about 9 GB left for the KV cache; the 16-bit one does not fit.

Common mistakes

  • Comparing one host's quantized model with another host's full-precision model as if they were the same.
  • Relying on the original model's benchmark scores for a heavily quantized version.
  • Forgetting that the KV cache for long contexts needs memory on top of the weights.

See it on a real card: Where to run it on Llama 4 Maverick.

Related terms