Sampling (top-p and top-k)
How the next token is picked from the model's probabilities. Top-k keeps only the k most likely tokens, top-p (nucleus sampling) keeps the smallest set whose probabilities add up to p, and one token is then drawn at random from what remains.
Why it matters. Together with temperature, these settings control how varied or predictable the output is.
How it works
At each step the model gives a probability for every token in its vocabulary. Greedy decoding always takes the most likely token; sampling draws at random after optional filtering. Temperature reshapes the distribution (lower is sharper), top-k cuts it to the k most likely tokens, and top-p cuts it to the most likely tokens whose probabilities together reach p. Some engines also offer min-p, which drops tokens far less likely than the top one.
Support varies: some APIs expose temperature and top-p but not top-k, some reasoning models fix or reject sampling parameters, and some makers advise changing temperature or top-p but not both. Open-weights models often come with recommended settings in their model card, and serving engines' defaults may differ from them.
For extraction, classification and code, use low temperature or the maker's recommended settings; for more variety, raise them. Even at temperature 0, outputs are not guaranteed to be identical across calls, because batching and floating-point effects can change results.
Worked example
Illustrative: next-token probabilities are A 0.50, B 0.25, C 0.15, D 0.06 and E 0.04. With top-k = 2, only A and B remain, renormalized to about 0.67 and 0.33. With top-p = 0.9, A, B and C remain (0.50 + 0.25 + 0.15 = 0.90), renormalized to about 0.56, 0.28 and 0.17.
Common mistakes
- Copying sampling settings between models, when makers recommend different values and some reasoning models ignore or reject them.
- Expecting temperature 0 to give byte-identical output on every call.
- Raising temperature and top-p together without testing, which can tip output into incoherence.