thelastapi
Get started free

How to handle rate limits on LLM APIs

Avoid Claude API rate limit errors by handling Retry-After headers, managing token throughput, and configuring daily spending limits.

On this page 6 sections

To avoid rate limit errors when calling Claude models, implement exponential backoff using the Retry-After header, throttle concurrent calls to your key tier, and raise daily spend or project budgets before reaching capacity. The gateway exposes models like Claude Sonnet 5.5 (vibiz/smart), Claude Opus 5.5 (vibiz/pro), and Claude Haiku 5.5 (vibiz/smart-mini), as well as lab identifiers like anthropic/claude-opus-5.5. When a call is refused with an HTTP 429 status code, the response payload indicates whether the restriction is caused by request volume, token throughput, or daily financial caps.

Understanding the precise trigger for an error prevents your code from making redundant calls that deplete quotas. For an overview of common status codes across providers, see our guide on LLM API errors explained: 429, 503 and context length.

What triggers rate limit errors on Claude models?

Rate limits are enforced at two distinct levels: operational throughput (requests, tokens, and concurrency) and financial ceilings (daily account caps and project budgets). When you send a request to /v1/ai/chat/completions, the platform evaluates your usage against these parameters.

If your request exceeds any limit, the gateway answers with an HTTP 429 and a JSON error object containing details.reason.

details.reasonCategoryDefault thresholdPrimary resolution
requests_per_minuteThroughput120 (project key) / 300 (account key)Pause using the duration in Retry-After
tokens_per_minuteThroughput200k (project key) / 400k (account key)Lower max_tokens or queue inputs
concurrencyThroughput25 in flight (project) / 50 in flight (account); a call over it waits up to 60 s for a slot firstLimit active parallel asynchronous requests
daily_spendFinancial$5 per account per UTC dayRaise safety cap in Billing or via CLI
project_budgetFinancial$1 per project per UTC dayRaise project daily budget in project settings
end_user_budgetFinancialApp-defined via end_user_daily_budget_usdAdjust individual end-user allocations

Throughput limits evaluate short rolling windows, while financial limits reset at midnight UTC (00:00:00Z).

How to inspect a 429 response

Every 429 response maintains an OpenAI-compatible error format. This allows standard client libraries to parse the failure cleanly:

JSON
{
  "error": {
    "code": "rate_limited",
    "message": "Rate limit reached.",
    "type": "requests",
    "param": null,
    "details": {
      "reason": "requests_per_minute"
    }
  }
}

When receiving this payload, check two items immediately:

  1. The Retry-After response header: When details.reason is requests_per_minute, tokens_per_minute, or concurrency, this header specifies the exact number of seconds to delay before repeating the call.
  2. The details.reason field: If the reason is daily_spend or project_budget, retrying the call without raising the cap will fail continuously until midnight UTC.

If you stream responses using server-sent events (stream: true), an in-flight rate limit arrives as a final data block: data: {"error": { ... }} without a trailing [DONE]. For design patterns that help automated tools process these messages, read our guide to agent-friendly APIs: errors that say what to do next.

How to handle retries with backoff in code

Automated clients should read Retry-After headers and apply jitter when scheduling retries. This prevents multiple concurrent workers from synchronizing their retry attempts.

Here is an example using the OpenAI SDK connected to the unified gateway via environment variables defined in AI docs:

TypeScript
import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: process.env.VIBIZ_AI_BASE_URL,
  apiKey: process.env.VIBIZ_AI_API_KEY,
});

async function callClaudeWithRetry(prompt: string, maxRetries = 3) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await openai.chat.completions.create({
        model: 'vibiz/smart',
        messages: [{ role: 'user', content: prompt }],
        max_tokens: 1024,
      });
    } catch (error: any) {
      if (error?.status === 429) {
        const reason = error?.error?.details?.reason;

        // Permanent failures for the current UTC day
        if (reason === 'daily_spend' || reason === 'project_budget') {
          throw new Error(`Execution halted: ${reason} limit reached. Raise budgets to continue.`);
        }

        if (attempt === maxRetries) {
          throw error;
        }

        // Extract Retry-After header or default to backoff with jitter
        const retryHeader = error?.headers?.['retry-after'];
        const delayMs = retryHeader
          ? parseInt(retryHeader, 10) * 1000
          : Math.pow(2, attempt) * 1000 + Math.random() * 500;

        await new Promise((resolve) => setTimeout(resolve, delayMs));
        continue;
      }
      throw error;
    }
  }
}

This pattern stops retries when budgets are depleted, avoiding unnecessary API consumption while resolving short-term throughput throttles. Learn more about unified configurations in our guide on one API key for GPT, Claude and Gemini.

How to raise project and account spending limits

When details.reason returns daily_spend or project_budget, calls cannot proceed until an owner increases the threshold or limits reset at midnight UTC. The platform sends automated warning emails at 80% and 100% of these limits, containing links directly to the setting that needs adjustment.

You can view current allocations from the CLI:

Terminal
vibiz usage

To modify these caps using the CLI or HTTP API, use a full-access key as outlined in Keys and environments:

Adjusting account daily safety caps

The account safety cap defaults to $5 per day to prevent runaway processes. Change it up to $1,000 using the CLI:

Terminal
vibiz billing limit 50

Or make an HTTP request to the billing endpoint:

Terminal
curl -X PATCH https://thelastapi.dev/v1/billing \
  -H "Authorization: Bearer $VIBIZ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"daily_spend_limit_usd": 50}'

Adjusting project AI budgets

Projects default to a $1 daily AI budget. Inspect and adjust this limit using:

Terminal
vibiz ai budget 10 --project prj_123

Or update the project configuration over HTTP:

Terminal
curl -X PATCH https://thelastapi.dev/v1/projects/prj_123/ai \
  -H "Authorization: Bearer $VIBIZ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"daily_budget_usd": 10}'

Setting daily_budget_usd to null removes the project budget restriction entirely, leaving only the account-level ceiling in effect. Review the Usage and limits documentation for details on limit notification settings.

How to structure workloads to stay under Claude throughput caps

To eliminate requests_per_minute and tokens_per_minute errors without relying solely on retry loops, manage your application's concurrency and token reservations:

  1. Choose the appropriate key type: A project key (vz_ai_...) defaults to 120 requests and 200,000 tokens a minute with 25 concurrent requests in flight. It can be raised up to 600 requests, 2,000,000 tokens a minute, and 50 concurrent requests. Account keys (vz_sk_...) provide 300 requests and 400,000 tokens a minute with 50 concurrent requests.
  2. Explicitly pass max_tokens: When omitted, max_tokens defaults to 1,024 (up to 16,384 per call). Before dispatching each request, the gateway reserves balance based on its worst-case calculation. Specifying a realistic max_tokens ensures you do not trigger pre-execution balance failures or tie up available concurrency.
  3. Account for long prompts: Claude Haiku 5.5 (vibiz/smart-mini) applies higher tier pricing above 100,000 input tokens. When submitting large context windows, batch prompts sequentially rather than issuing multiple parallel requests against the token-per-minute cap.
  4. Throttle client-side queues: Restrict your application worker pools to stay below your key's concurrency limit (25 for standard project keys, 50 for account keys; a call over it waits up to 60 s for a slot before a 429).

To monitor live token volumes across tools, models, and end users, run:

Terminal
vibiz usage --period 24h --by model

To configure key quotas, raise spending thresholds, or set up automated credit replenishment, review our guides on AI and Billing and credits.

Frequently asked questions

What is the difference between a project key and an account key rate limit?

An account key allows 300 requests and 400,000 tokens per minute with 50 in-flight calls. A project key defaults to 120 requests and 200,000 tokens per minute with 25 in-flight calls, but an owner can raise its limits up to 600 requests, 2,000,000 tokens, and 50 in-flight calls.

Why do I get a 429 rate limit error when my request volume is low?

An HTTP 429 status code is returned for both throughput limits and spending caps. Inspect `error.details.reason`: values such as `daily_spend` (account ceiling) or `project_budget` (project ceiling) indicate a financial limit has been reached rather than a per-minute request spike.

At what time do daily spend limits reset?

All daily spend ceilings and project AI budgets reset at midnight UTC (00:00:00Z). The reset timestamp is also provided in usage payloads under `limits.resets_at`.

Can Claude model calls switch to another model automatically during rate limits?

No. thelastapi never substitutes another model when your requested model encounters rate limits or availability issues. The gateway returns an explicit error code so your application maintains predictable output and cost.