Rate limits

How it works

Rate limits cap API usage over time, commonly as requests per minute, input tokens per minute and output tokens per minute, and sometimes as daily or monthly spend. When you exceed one, the API rejects requests, usually with HTTP 429, until the window resets.

Limits usually depend on your account's usage tier, which rises with spend and history, and differ by model. Some providers count cached tokens differently toward limits, and batch traffic usually has separate limits. Response headers often report remaining capacity and reset times.

Check the limits for your tier before launch, especially tokens per minute if prompts are long. Retry with exponential backoff and jitter, queue work, and ask for higher limits ahead of expected growth.

Worked example

Illustrative: with a limit of 400,000 input tokens per minute and prompts of 20,000 tokens, you can send at most 20 requests per minute, even if your request limit is 1,000 per minute.

Common mistakes

• 1Retrying 429 errors immediately in a tight loop, which keeps you throttled.
• 2Watching requests per minute when long prompts hit the tokens-per-minute limit first.
• 3Load-testing on a high-tier account and launching on a lower-tier one.