# Why your AI gateway should never swap models silently

> Automatic failover in an LLM API gateway should never silently swap models because output drift, broken tool calls, and unexpected costs break production.

**An LLM API gateway should never swap models automatically behind your back because differing model architectures cause silent schema failures, context truncation, and unpredictable billing.** When an upstream provider goes down, a production system needs an explicit error so your application code can decide whether to retry, degrade functionality, or switch to a deliberate fallback. thelastapi enforces strict model determinism: the exact model you name in your request is the only model that ever runs.

Many multi-model proxies promise high availability through automatic failover. In practice, substituting models silently creates severe failure modes that are difficult to debug in production.

## Why automatic failover breaks production systems

Automatic failover sounds convenient when testing prompts in a playground, but it quickly introduces stability problems once you connect real users and agents.

### 1. Tool calling and structured outputs fail
Different labs train their models on distinct function-calling schemas and JSON grammars. A prompt and tool definition tuned to run reliably on Claude Sonnet 5.5 (`vibiz/smart`) can produce malformed JSON, markdown backticks, or hallucinated parameter names when routed to a different lab's model. If your gateway silently swaps models during an upstream timeout, your downstream parser crashes.

### 2. Context window discrepancies
Context limits vary sharply across model tiers. For example, DeepSeek V4.1 Flash (`vibiz/lite`) and Gemini 3.8 Flash (`vibiz/balanced`) both support a 1,000,000-token context window. If a proxy silently redirects a 300,000-token prompt to a model with a 128,000-token limit, the call fails immediately with a context length error. Worse, some proxies truncate input text without warning, dropping necessary instructions or retrieval context.

### 3. Prompt behavior and reasoning drift
Different models follow instructions differently. A reasoning model like GPT-6.1 Terra (`vibiz/reasoning`) evaluates step-by-step logic before answering. If your gateway redirects a query from a reasoning model to a small chat model during peak load, the quality of the answer degrades immediately. Your users receive inconsistent responses, and your evaluation benchmarks become meaningless.

### 4. Unpredictable token pricing
Model prices span several orders of magnitude. Routing traffic away from an inexpensive model to a larger model can spike your infrastructure spend overnight. Conversely, dropping from a premium model to a budget tier to avoid downtime degrades output quality for users who paid for high-end performance. For a deeper breakdown of pricing and architectural differences across labs, see our guide on [using OpenAI directly versus a model gateway](https://thelastapi.dev/blog/openai-direct-vs-model-gateway).

## How thelastapi handles model downtime

thelastapi provides an OpenAI-compatible interface at `/v1/ai`. Rather than concealing upstream issues with automatic swaps, thelastapi returns clear, standard HTTP status codes and error payloads so your code can react.

If a requested model is unavailable, the gateway returns an explicit failure:

- `model_unavailable` (HTTP 503): Returned when no route can run that specific model at that moment. The response includes `details.retry_after_seconds` so your system knows how long to pause before trying again.
- `provider_unavailable` (HTTP 503): Returned when every connection route to that lab has stalled or timed out (`details.reason` reports `timeout`, `stall`, or `circuit_open`).
- `invalid_request` (HTTP 400): If you pass an unknown model ID or make a typo, the gateway rejects it immediately with `param: "model"`. It will never default to a random live model.

When streaming completions via Server-Sent Events (SSE), errors that occur mid-generation arrive as a final event formatted as `data: {"error": {...}}` without a closing `[DONE]` tag.

Every JSON response from the gateway confirms exactly what executed:
- `served_model`: Names the precise model that generated the completion. (On speech endpoints, this is provided in the `X-Vibiz-Served-Model` HTTP header.)
- `cost_micro_usd`: States the exact cost of that specific call in micro-USD.

The table below outlines the standard model tiers available on thelastapi:

| thelastapi model | Runs on | Best suited for | List price per million tokens (in / out) |
| --- | --- | --- | --- |
| `vibiz/fast` | GPT-6 Luna (OpenAI) | Everyday chat, tool calls, JSON | $0.10 / $0.50 |
| `vibiz/lite` | DeepSeek V4.1 Flash (DeepSeek) | Budget tasks, 1M context | $0.14 / $0.42 |
| `vibiz/smart-mini` | Claude Haiku 5.5 (Anthropic) | Quick, strong writing and code | $0.10 / $0.50 |
| `vibiz/balanced` | Gemini 3.8 Flash (Google) | Balanced quality, 1M context | $0.825 / $4.125 |
| `vibiz/reasoning` | GPT-6.1 Terra (OpenAI) | Deep reasoning, math, planning | $1.00 / $5.00 |
| `vibiz/smart` | Claude Sonnet 5.5 (Anthropic) | Coding, writing, agent workflows | $2.00 / $10.00 |
| `vibiz/pro` | Claude Opus 5.5 (Anthropic) | Complex chat, long agent loops | $4.00 / $20.00 |

Beyond these aliases, you can invoke more than 400 models directly by their provider ID, such as `moonshotai/kimi-k3.1-pro` or `anthropic/claude-opus-5.5`. Read more in [one API key for GPT, Claude and Gemini](https://thelastapi.dev/blog/one-api-key-gpt-claude-gemini).

## How to handle failover cleanly in your application

When failover logic lives inside your application instead of a black-box gateway, you control the recovery path. Your application can choose a verified alternative model, adjust the prompt, or return a graceful message to the user.

Because thelastapi is fully OpenAI-compatible, you can use the standard OpenAI SDK:

```ts
import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: process.env.VIBIZ_AI_BASE_URL,
  apiKey: process.env.VIBIZ_AI_API_KEY,
});

async function runChat(prompt: string) {
  try {
    // Attempt the primary model first
    return await openai.chat.completions.create({
      model: 'vibiz/smart',
      messages: [{ role: 'user', content: prompt }],
    });
  } catch (err: any) {
    // Check if the specific model or provider is down
    if (err?.code === 'model_unavailable' || err?.code === 'provider_unavailable') {
      console.warn('Primary model unavailable. Invoking intentional fallback.');
      
      // Deliberately fall back to a model tested with your application
      return await openai.chat.completions.create({
        model: 'vibiz/fast',
        messages: [{ role: 'user', content: prompt }],
      });
    }
    throw err;
  }
}
```

In an application built with [@vibiz/app](https://thelastapi.dev/docs/app-sdk), initializing the client uses `vibiz.ai`:

```ts
import { vibiz } from '@vibiz/app';
import OpenAI from 'openai';

const openai = new OpenAI(vibiz.ai);

const res = await openai.chat.completions.create({
  model: 'vibiz/fast',
  messages: [{ role: 'user', content: 'Extract key terms from this invoice.' }],
});

console.log(res.served_model);
console.log(res.cost_micro_usd);
```

You can test model listings and check prices directly from your terminal using the CLI:

```bash
vibiz ai models kimi
vibiz ai "Summarize this diff" --model vibiz/smart
```

## What happens when using your own Google key

thelastapi lets you bring your own Gemini API key for Google models. When configured, Google models run directly on your own Google account at no platform markup.

```bash
GEMINI_API_KEY=your_key vibiz ai keys set google
```

The gateway enforces a strict policy here as well: **there is no silent fallback**. If you configure your own key and Google returns an error or experiences downtime, the call fails immediately with `provider_failed`, `customer_key_rejected`, or `provider_unavailable`. It is never retried against thelastapi's accounts behind your back, preventing surprise charges on your thelastapi credit balance. For details, see [using your own Gemini API key through a gateway](https://thelastapi.dev/blog/use-your-own-gemini-key).

## Budget limits that prevent automated billing cascades

When client applications implement automatic retry or failover logic, a bug can trigger runaway loops that consume credits rapidly. thelastapi provides hard spend limits to keep your project safe:

- **Account daily spend ceiling**: Set to $5 per UTC day by default across AI and actions. An owner can raise this in the console or via `PATCH /v1/billing {"daily_spend_limit_usd": 50}` (up to $1,000). When reached, the gateway returns `429 rate_limited` with `details.reason: "daily_spend"`.
- **Project daily budget**: Set to $1 per project per UTC day by default. The owner can adjust this via `PATCH /v1/projects/:id/ai {"daily_budget_usd": 10}`. Exceeding this limit returns `429` with `details.reason: "project_budget"`.
- **Per-request pre-flight check**: Each call's worst-case token cost is calculated before execution. If the account balance cannot support the worst case (plus a $0.50 margin), the call is refused with `402 payment_required`.
- **Concurrency and rate caps**: Project keys are capped at 120 requests and 200,000 tokens per minute (with 25 concurrent requests in flight), while account keys allow 300 requests and 400,000 tokens per minute.

Review the full specifications in the [AI docs](https://thelastapi.dev/docs/ai) and the [Usage and limits](https://thelastapi.dev/docs/usage) documentation.

To start building deterministic, transparent AI pipelines without surprise model swaps, check your project keys with `vibiz keys` or create your account at [thelastapi.dev/start](https://thelastapi.dev/start).

## Frequently asked questions

### Why is automatic model swapping harmful in an LLM API gateway?

Automatic swapping causes unexpected formatting errors in tool calls, context window truncation, output quality drift, and unpredictable cost spikes. When gateways swap models silently, downstream application logic and parsers frequently fail.

### How does thelastapi inform my application when a model is down?

Instead of routing your request to a different model, the gateway returns an HTTP 503 error with code `model_unavailable` or `provider_unavailable`. Each response also contains `details.retry_after_seconds` or a specific failure reason like `timeout` or `circuit_open`.

### How can I verify which model actually executed my prompt?

Every JSON response includes the `served_model` field containing the exact model identifier that fulfilled the request, alongside `cost_micro_usd`, which states the exact price of the call. Audio speech endpoints return the served model in the `X-Vibiz-Served-Model` HTTP header.

### If I supply my own Gemini API key, will calls fall back to thelastapi when Google fails?

No. Calls mapped to your Google key will fail with `customer_key_rejected` or `provider_unavailable` if Google rejects the key or goes down. They are never retried on thelastapi's shared accounts behind your back.

---

Published 2026-11-15 · https://thelastapi.dev/blog/no-silent-model-fallback
