# Do AI API gateways log your prompts? Questions to ask first

> LLM API aggregators record operational metrics for billing, but whether they store raw prompt text depends on their logging design and settings.

**Most LLM API aggregators log operational telemetry like token counts and latency, but whether they store your raw prompt text depends on their architecture and logging configuration.** Some aggregators retain full prompt and completion payloads by default to support debugging consoles, prompt caching, or evaluations. Other gateways act as stateless routing proxies, discarding text payloads once streamed and recording only the token units needed to meter usage and bill your account.

Before you route sensitive production traffic, proprietary code, or user chats through a third-party gateway, you should understand what data is retained, how models execute, and what happens when an upstream provider goes down.

## What data does an LLM API gateway collect?

Every AI API gateway must collect telemetry to run its service. At a minimum, a gateway needs accounting metrics to calculate bills, enforce rate limits, and track system health. That operational metadata is separate from the contents of your prompt.

Here is how data collection breaks down across gateways:

| Data category | Examples | Why gateways track it |
| --- | --- | --- |
| **Usage metadata** | Token counts (`input_tokens`), timestamps, latency, HTTP status, model id | Billing calculations, rate limiting, and accounting |
| **Identity metadata** | Project id, API key id, user identifier (`user` field) | Attributing spend to projects, environments, or end users |
| **Payload data** | System prompts, chat message arrays, image input URLs | Debugging, prompt caching, evaluation, or playground review |
| **Generated outputs** | Assistant text, generated images, synthesized speech audio | Delivering the API response and permanent asset hosting |

With thelastapi, usage tracking at `/v1/ai` records only billing units. Every billable unit—such as an AI call's tokens or an action—is recorded once with what it cost to run (`cost`) and what the account is charged (`billed`). You can inspect these metrics at any time using the CLI:

```bash
vibiz usage --period 7d --by model
vibiz usage --by end_user --project prj_123
```

A gateway does not need to store your conversational text in an analytics database just to count your tokens and charge your balance. You can review all metered units in the [Usage and limits](https://thelastapi.dev/docs/usage) documentation.

## Five questions to ask before routing prompts to a gateway

When you evaluate any unified AI interface or LLM aggregator, ask the provider these five questions:

### 1. Does the gateway store raw prompt text or only token metrics?
Find out whether the service maintains a database of user prompts. Some platforms keep text logs enabled by default to power hosted dashboards, user replay tools, or prompt playgrounds. If you handle private user data, healthcare information, or proprietary code, verify that text chat requests do not persist in request-log stores unless you explicitly choose to store them.

### 2. Does the gateway silently swap models when one fails?
Some aggregators implement automatic fallbacks. If an upstream lab suffers an outage or rate limit, the aggregator re-routes the prompt to a different model from another vendor to keep the request alive. 

This creates immediate operational risks:
- Prompts engineered for Claude Opus 5.5 may produce broken JSON or unhelpful results if routed to a smaller model.
- Token pricing varies widely across tiers.
- Compliance agreements that cover one lab may not apply to another.

thelastapi never swaps models quietly behind your back. If the requested model is down, the API returns a `503` status with `model_unavailable` or `provider_unavailable` and a retry hint (`details.retry_after_seconds`). The model you specify in your request is strictly the model that runs.

### 3. Does the API return the exact model that answered?
Every response should tell you what executed upstream. For chat completions through `POST /v1/ai/chat/completions`, the JSON response from thelastapi includes the `served_model` field and the exact call cost in `cost_micro_usd`. For audio speech generation, the gateway sets the `X-Vibiz-Served-Model` HTTP response header.

### 4. How does the provider handle media prompts versus chat prompts?
Text chat completions are often transient, but media generation (images, speech, video) produces binary files that must live somewhere. Check whether generated files expire immediately or are saved to your account.

### 5. Can you use your own provider keys?
If you have preexisting credits with a provider like Google, check whether the gateway allows direct routing. With thelastapi, you can register your Google Gemini key using `vibiz ai keys set google`. When configured, Google models run directly on your own Google account at no charge on thelastapi (`cost: 0`), while calls to models from other labs continue using your standard account credits.

## Media prompts versus chat prompts: how files are stored

Text completions and media requests work differently when it comes to storage. While text chat messages pass through the `/v1/ai/chat/completions` gateway, generated media requires storage space for the resulting assets.

In thelastapi, generated files are handled according to your project context:

- **Requests with a project key:** Every image, video, and audio file you create is saved to the project's Files. The record includes the generated file, the model name, the prompt, and the cost. File names follow a clear structure based on the date and prompt text (such as `2026-10-28-cold-email-cover.png`). You receive a permanent URL in the response payload.
- **Requests without a project:** When a call runs under an account key with no project attached, the asset cannot be stored in a project directory. The file link remains active for 24 hours, after which it expires.

Here is how you generate an image from the command line:

```bash
vibiz image "a croissant on a marble table" -o hero.png
```

To learn more about calling multiple labs through a single gateway interface, see [how a unified AI API works](https://thelastapi.dev/blog/one-api-key-gpt-claude-gemini).

## How to verify the model and per-call cost in code

The gateway runs at `https://thelastapi.dev/v1/ai` and is fully compatible with the standard OpenAI SDK. You can inspect the returned model and cost metadata directly in your application code:

```ts
import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: process.env.VIBIZ_AI_BASE_URL,
  apiKey: process.env.VIBIZ_AI_API_KEY,
});

async function runPrompt() {
  const response = await openai.chat.completions.create({
    model: 'vibiz/fast',
    messages: [{ role: 'user', content: 'Draft a three-line release note.' }],
  });

  console.log('Answer:', response.choices[0]?.message.content);

  // Inspect the gateway metadata fields returned in the JSON payload
  const rawResponse = response as typeof response & {
    served_model: string;
    cost_micro_usd: number;
  };
  console.log('Served model:', rawResponse.served_model);
  console.log('Cost in micro-USD:', rawResponse.cost_micro_usd);
}

runPrompt();
```

You can choose models by their stable role tier or by their raw lab identifier:

| thelastapi model | Upstream lab model | Good for | Price per million tokens (in / out) |
| --- | --- | --- | --- |
| `vibiz/fast` | GPT-6 Luna (OpenAI) | Chat, tool calling, JSON mode | $0.10 / $0.50 |
| `vibiz/lite` | DeepSeek V4.1 Flash | Low-cost high-volume chat | $0.14 / $0.42 |
| `vibiz/smart-mini` | Claude Haiku 5.5 (Anthropic) | Fast writing and code tasks | $0.10 / $0.50 |
| `vibiz/balanced` | Gemini 3.8 Flash (Google) | Balanced speed and quality | $0.825 / $4.125 |
| `vibiz/smart` | Claude Sonnet 5.5 (Anthropic) | Complex coding and agents | $2.00 / $10.00 |
| `vibiz/pro` | Claude Opus 5.5 (Anthropic) | Long agent sessions | $4.00 / $20.00 |

To optimize token spending across these tiers, read [how to cut your LLM API bill](https://thelastapi.dev/blog/cut-llm-api-bill).

## How to cap spend and audit usage across projects

Beyond inspecting logs, you need hard boundaries to prevent runaways or excessive API spend. thelastapi enforces limits at three distinct levels:

1. **Daily account safety cap:** Every account has a default $5 daily cap covering AI and actions combined. Account owners can raise this in Billing or via `PATCH /v1/billing {"daily_spend_limit_usd": 50}`. When hit, calls return a `429` status with `details.reason: "daily_spend"`.
2. **Project AI budget:** Each project defaults to a $1 daily AI budget. The owner can adjust this in the console or via `PATCH /v1/projects/:id/ai {"daily_budget_usd": 10}`. When hit, the gateway answers `429` with `details.reason: "project_budget"`.
3. **End-user budget:** Pass a stable pseudonymous user id via the `user` field in chat completions or the `X-Vibiz-User` HTTP header. The owner can set `end_user_daily_budget_usd` on the project to cap what any single end user can spend per day.

For details on configuring client caps, see [per-user AI budgets](https://thelastapi.dev/blog/per-user-ai-budgets).

Next: review the full list of supported chat, image, voice, and reasoning models on the [AI docs page](https://thelastapi.dev/docs/ai), or use `vibiz ai models` in your terminal to search available endpoints and their list prices.

## Frequently asked questions

### Do all LLM aggregators log prompt text by default?

No, aggregators differ in how they handle payloads. Some maintain persistent databases of prompt and completion text for evaluation dashboards, while others operate as stateless proxies that only record token counts and billing metadata.

### How can I tell which lab model actually answered my API call?

In thelastapi, every chat completion response includes the `served_model` field in its JSON body, and audio requests return an `X-Vibiz-Served-Model` header. If a model is unavailable, the gateway returns a 503 error instead of silently routing your prompt to an alternative provider.

### Where are prompts stored when generating images or audio?

When you generate media within a project, the prompt, model id, cost, and output file are saved to your project's Files with a permanent link. If you generate media using an account key outside a project, files are not saved to Files and their temporary links expire after 24 hours.

### What happens if an LLM model provider experiences an outage?

thelastapi returns an HTTP 503 error with either `model_unavailable` or `provider_unavailable` alongside a retry hint. The gateway never falls back to an unrequested model behind your back.

---

Published 2026-11-15 · https://thelastapi.dev/blog/do-ai-gateways-log-prompts
