Guides

Rate limits and timeouts

How many requests and tokens a key can send each minute, how many can run at once, how large a request can be, and how long replies can run.

On this page

Rate limits

Each API key has two limits of its own:

LimitPer keyWhat counts
Requests600 a minuteEvery request to chat completions, Responses, Messages and count tokens.
Tokens2,000,000 a minuteEach request’s estimated input, plus its max_tokens in full.
  • The token limit counts the longest reply you allowed, not the reply you got, because it’s checked before the model runs. A request with no max_tokens counts 16,000 for its reply. Setting max_tokens close to what you need lets more requests through.
  • Input is estimated from the size of the request, at about four bytes per token.
  • Counts start again at the beginning of each minute, rather than over a rolling 60 seconds.

When you hit a limit

Over either limit, the request isn’t run and you get a 429 with the code rate_limit_exceeded. Its type says which limit: requests or tokens. The retry-after header gives the whole seconds until the minute ends.

Response
HTTP/1.1 429 Too Many Requests
retry-after: 37
x-request-id: req_0b7d5e2c91f84a6e3d10c9b7a52e8f46

{
  "error": {
    "message": "Rate limit reached. Try again shortly",
    "type": "tokens",
    "param": null,
    "code": "rate_limit_exceeded"
  }
}

A busy model also answers 429, with type server_error and a retry-after of 2. Rate limited requests show on the Usage page as not run, so you can see when they happened. See Retrying for a retry loop.

Size limits

WhatLimitOver it
Request body, images included8 MiB413
stop or stop_sequences4 sequences400
Reply lengthmax_tokens, 16,000 if not set, never more than the model allowsThe reply stops, as it reached its limit
A streamed reply32 MiB of textThe stream ends with finish_reason: "length"
Idempotency-Key128 characters400

Timeouts and long replies

How long a model has to start answering, and to finish, depends on the model. If it doesn’t answer in time, you get a 502 with server_error, which is worth retrying.

  • For long replies, use stream. Text arrives as it’s written, and after 15 seconds with nothing to send the stream carries a keep-alive, so quiet connections aren’t dropped along the way.
  • Without streaming, nothing comes back until the reply is complete. Give your HTTP client a timeout of several minutes for long replies. The OpenAI and Anthropic SDKs wait up to 10 minutes by default.
  • Closing the connection stops the request. If no reply had started, nothing is charged. If a stream was under way, you’re charged for what was produced up to then, and the request shows as stopped on the Usage page.

The OpenAI and Anthropic SDKs retry timed-out requests on their own. Send an Idempotency-Key so a retry can’t run twice.