Handling LLM API Errors, Rate Limits and Retries

LLM APIs are network services under heavy load, so failures are normal. The trick is knowing which errors to retry and which to surface to the user.

Retryable vs non-retryable

• Retry: 429 rate limit, 5xx server errors, overloaded errors, timeouts and dropped connections.
• Don't retry: 400 bad request, 401 authentication, 403 permission, 404 not found. Retrying won't change the answer; fix the request.

Built-in retries

The official SDKs already retry connection errors, 408, 409, 429 and 5xx with exponential backoff (2 retries by default). You can tune this:

const client = new Anthropic({ maxRetries: 4, timeout: 120_000 }); // timeout in ms in TypeScript

client = anthropic.Anthropic(max_retries=4, timeout=120.0)  # seconds in Python

Catch typed errors, most specific first

try {
  const res = await client.messages.create(params);
} catch (err) {
  if (err instanceof Anthropic.RateLimitError) {
    return queueForLater(params);          // still 429 after the SDK's retries
  } else if (err instanceof Anthropic.BadRequestError) {
    logger.error('Invalid request', err.message); // fix the code, don't retry
    throw err;
  } else if (err instanceof Anthropic.APIError) {
    logger.error(`API error ${err.status}`, err.message);
    throw err;
  }
  throw err; // network or unknown errors
}

Check stop reasons, too

A successful HTTP 200 can still be an incomplete answer:

switch (res.stop_reason) {
  case 'max_tokens':
    // Output was cut off: raise max_tokens or ask for a shorter answer
    break;
  case 'refusal':
    // The model declined; show a friendly message instead of empty content
    break;
}

Rate limit hygiene

• Limit concurrency with a queue (for example p-limit) instead of firing hundreds of requests at once.
• Read the retry-after header when you implement your own retry loop.
• Cache identical requests; the cheapest call is the one you don't make.