aiaimodel.cards
Weights & training50

Pretraining

Large-scale next-token training on broad data

Pretraining

The first and largest training stage, in which a model learns to predict the next token over a very large corpus of text, and often code, images or audio. It produces a base model.

Why it matters. Pretraining data, and the date it ends, determine most of what a model knows.

How it works

In pretraining the model sees a very large number of tokens and is trained to predict each next one; along the way it picks up language, facts, code and reasoning patterns. It takes most of the training compute, and the result is a base model that continues text rather than following instructions.

Makers disclose pretraining data in varying detail: sources (public web, licensed, synthetic), filtering and sometimes size, often in a paragraph of the model or system card. The knowledge cutoff marks roughly where the training data ends, and coverage of the last months before it is usually thin.

The knowledge cutoff is a data date, not a release date; models often ship months after it. Knowledge from pretraining is fixed, so anything newer has to come in through tools, retrieval or the prompt.

Worked example

Illustrative: a dense 8B-parameter model pretrained on 15 trillion tokens needs roughly 6 x parameters x tokens = 6 x 8 x 10^9 x 15 x 10^12 = 7.2 x 10^23 operations, using a common rule of thumb for training compute.

Common mistakes

  • Treating the knowledge cutoff as the release date, or as the date up to which knowledge is complete.
  • Assuming a list of disclosed data sources tells you whether specific content was included.
  • Expecting a pretrained-only base checkpoint to follow instructions like a chat model.

Related terms