Batch API

How it works

A batch API accepts many requests at once, processes them asynchronously and returns the results later, within a completion window the provider states. In exchange for giving up immediate answers you pay a discounted rate, often about half the standard price.

Providers differ in completion windows, batch size limits, supported features and whether the batch discount combines with prompt caching. Batch traffic usually has its own rate limits, and results can come back in any order, so each request carries an ID you match on.

Batch suits evaluations, classification, document processing, embedding backfills and nightly jobs. It does not suit anything a user is waiting on.

Worked example

Illustrative: classifying 100,000 support tickets at 1,000 input and 50 output tokens each, with $1 per 1M input and $5 per 1M output, costs $100 + $25 = $125 in real time. At a 50% batch discount it is about $62.50.

Common mistakes

• 1Matching results to inputs by position instead of by request ID.
• 2Designing a pipeline that assumes batches finish in minutes rather than within the full completion window.
• 3Not handling individual failed or expired requests inside an otherwise successful batch.