Inference

How it works

Inference uses a model whose weights are fixed: the prompt is split into tokens, passed through the network, and the model produces output tokens one at a time. The model does not learn from your requests while answering them; whether a provider later uses your data for training is a separate policy question.

For language models, inference has two phases, prefill (reading the prompt) and decode (generating output tokens), with different costs and speeds. Providers run it on clusters of GPUs or other accelerators, using serving engines that batch many users' requests together, which is why price per token falls with scale and speed varies with load.

API users see inference as per-token prices, rate limits, latency and output speed. Self-hosting means taking on the hardware, the serving engine and capacity planning, and the cost per token then depends on how busy you keep the hardware.

Worked example

Illustrative: a request with a 2,000-token prompt and a 500-token answer at $3 input and $15 output per 1M tokens costs 2,000 / 1,000,000 x $3 + 500 / 1,000,000 x $15 = $0.006 + $0.0075 = $0.0135. The shorter output is the larger share of the bill.

Common mistakes

• 1Assuming the model learns from your conversations as you use it; memory across requests exists only if your app or the provider adds it.
• 2Estimating cost from input tokens alone, when output tokens are usually priced several times higher.
• 3Planning capacity from average traffic; inference load is bursty, and peaks set the rate limits and hardware you need.