Parameters

How it works

Every connection in a neural network has a numeric value learned in training. In a language model most parameters sit in the attention and feed-forward matrices, plus the table that maps tokens to vectors. Counts are given in billions (B) or trillions (T).

Many closed models' parameter counts are not disclosed, and third-party estimates are guesses. Open-weights makers state them, and for mixture-of-experts models they usually give both total and active counts. Some counts include the embedding table and some don't, so the same model can be quoted slightly differently.

Memory scales with total parameters x bytes per parameter, and compute per token with active parameters. Capability also depends on training data, compute, post-training and architecture, so a newer smaller model can beat an older larger one.

Worked example

Illustrative: an 8B-parameter model needs 8 x 2 = 16 GB for its weights at BF16 and a 70B model 140 GB. A forward pass costs roughly 2 x parameters operations per token, about 16 billion for the 8B model and 140 billion for the 70B one.

Common mistakes

• 1Treating parameter count as a quality ranking across model families or generations.
• 2Quoting a parameter count for a closed model from rumor or third-party estimates.
• 3Sizing a mixture-of-experts model's memory from its active parameter count.