Guides
Rate limits and timeouts
How many requests and tokens a key can send each minute, how many can run at once, how large a request can be, and how long replies can run.
On this page
Rate limits
Each API key has two limits of its own:
| Limit | Per key | What counts |
|---|---|---|
| Requests | 600 a minute | Every request to chat completions, Responses, Messages and count tokens. |
| Tokens | 2,000,000 a minute | Each request’s estimated input, plus its max_tokens in full. |
- The token limit counts the longest reply you allowed, not the reply you got, because it’s checked before the model runs. A request with no
max_tokenscounts 16,000 for its reply. Settingmax_tokensclose to what you need lets more requests through. - Input is estimated from the size of the request, at about four bytes per token.
- Counts start again at the beginning of each minute, rather than over a rolling 60 seconds.
When you hit a limit
Over either limit, the request isn’t run and you get a 429 with the code rate_limit_exceeded. Its type says which limit: requests or tokens. The retry-after header gives the whole seconds until the minute ends.
HTTP/1.1 429 Too Many Requests
retry-after: 37
x-request-id: req_0b7d5e2c91f84a6e3d10c9b7a52e8f46
{
"error": {
"message": "Rate limit reached. Try again shortly",
"type": "tokens",
"param": null,
"code": "rate_limit_exceeded"
}
}A busy model also answers 429, with type server_error and a retry-after of 2. Rate limited requests show on the Usage page as not run, so you can see when they happened. See Retrying for a retry loop.
Size limits
| What | Limit | Over it |
|---|---|---|
| Request body, images included | 8 MiB | 413 |
stop or stop_sequences | 4 sequences | 400 |
| Reply length | max_tokens, 16,000 if not set, never more than the model allows | The reply stops, as it reached its limit |
| A streamed reply | 32 MiB of text | The stream ends with finish_reason: "length" |
Idempotency-Key | 128 characters | 400 |
Timeouts and long replies
How long a model has to start answering, and to finish, depends on the model. If it doesn’t answer in time, you get a 502 with server_error, which is worth retrying.
- For long replies, use
stream. Text arrives as it’s written, and after 15 seconds with nothing to send the stream carries a keep-alive, so quiet connections aren’t dropped along the way. - Without streaming, nothing comes back until the reply is complete. Give your HTTP client a timeout of several minutes for long replies. The OpenAI and Anthropic SDKs wait up to 10 minutes by default.
- Closing the connection stops the request. If no reply had started, nothing is charged. If a stream was under way, you’re charged for what was produced up to then, and the request shows as stopped on the Usage page.
The OpenAI and Anthropic SDKs retry timed-out requests on their own. Send an Idempotency-Key so a retry can’t run twice.