thelastapi
Get started free

Do AI API gateways log your prompts? Questions to ask first

LLM API aggregators record operational metrics for billing, but whether they store raw prompt text depends on their logging design and settings.

On this page 6 sections

Most LLM API aggregators log operational telemetry like token counts and latency, but whether they store your raw prompt text depends on their architecture and logging configuration. Some aggregators retain full prompt and completion payloads by default to support debugging consoles, prompt caching, or evaluations. Other gateways act as stateless routing proxies, discarding text payloads once streamed and recording only the token units needed to meter usage and bill your account.

Before you route sensitive production traffic, proprietary code, or user chats through a third-party gateway, you should understand what data is retained, how models execute, and what happens when an upstream provider goes down.

What data does an LLM API gateway collect?

Every AI API gateway must collect telemetry to run its service. At a minimum, a gateway needs accounting metrics to calculate bills, enforce rate limits, and track system health. That operational metadata is separate from the contents of your prompt.

Here is how data collection breaks down across gateways:

Data categoryExamplesWhy gateways track it
Usage metadataToken counts (input_tokens), timestamps, latency, HTTP status, model idBilling calculations, rate limiting, and accounting
Identity metadataProject id, API key id, user identifier (user field)Attributing spend to projects, environments, or end users
Payload dataSystem prompts, chat message arrays, image input URLsDebugging, prompt caching, evaluation, or playground review
Generated outputsAssistant text, generated images, synthesized speech audioDelivering the API response and permanent asset hosting

With thelastapi, usage tracking at /v1/ai records only billing units. Every billable unit—such as an AI call's tokens or an action—is recorded once with what it cost to run (cost) and what the account is charged (billed). You can inspect these metrics at any time using the CLI:

Terminal
vibiz usage --period 7d --by model
vibiz usage --by end_user --project prj_123

A gateway does not need to store your conversational text in an analytics database just to count your tokens and charge your balance. You can review all metered units in the Usage and limits documentation.

Five questions to ask before routing prompts to a gateway

When you evaluate any unified AI interface or LLM aggregator, ask the provider these five questions:

1. Does the gateway store raw prompt text or only token metrics?

Find out whether the service maintains a database of user prompts. Some platforms keep text logs enabled by default to power hosted dashboards, user replay tools, or prompt playgrounds. If you handle private user data, healthcare information, or proprietary code, verify that text chat requests do not persist in request-log stores unless you explicitly choose to store them.

2. Does the gateway silently swap models when one fails?

Some aggregators implement automatic fallbacks. If an upstream lab suffers an outage or rate limit, the aggregator re-routes the prompt to a different model from another vendor to keep the request alive.

This creates immediate operational risks:

  • Prompts engineered for Claude Opus 5.5 may produce broken JSON or unhelpful results if routed to a smaller model.
  • Token pricing varies widely across tiers.
  • Compliance agreements that cover one lab may not apply to another.

thelastapi never swaps models quietly behind your back. If the requested model is down, the API returns a 503 status with model_unavailable or provider_unavailable and a retry hint (details.retry_after_seconds). The model you specify in your request is strictly the model that runs.

3. Does the API return the exact model that answered?

Every response should tell you what executed upstream. For chat completions through POST /v1/ai/chat/completions, the JSON response from thelastapi includes the served_model field and the exact call cost in cost_micro_usd. For audio speech generation, the gateway sets the X-Vibiz-Served-Model HTTP response header.

4. How does the provider handle media prompts versus chat prompts?

Text chat completions are often transient, but media generation (images, speech, video) produces binary files that must live somewhere. Check whether generated files expire immediately or are saved to your account.

5. Can you use your own provider keys?

If you have preexisting credits with a provider like Google, check whether the gateway allows direct routing. With thelastapi, you can register your Google Gemini key using vibiz ai keys set google. When configured, Google models run directly on your own Google account at no charge on thelastapi (cost: 0), while calls to models from other labs continue using your standard account credits.

Media prompts versus chat prompts: how files are stored

Text completions and media requests work differently when it comes to storage. While text chat messages pass through the /v1/ai/chat/completions gateway, generated media requires storage space for the resulting assets.

In thelastapi, generated files are handled according to your project context:

  • Requests with a project key: Every image, video, and audio file you create is saved to the project's Files. The record includes the generated file, the model name, the prompt, and the cost. File names follow a clear structure based on the date and prompt text (such as 2026-10-28-cold-email-cover.png). You receive a permanent URL in the response payload.
  • Requests without a project: When a call runs under an account key with no project attached, the asset cannot be stored in a project directory. The file link remains active for 24 hours, after which it expires.

Here is how you generate an image from the command line:

Terminal
vibiz image "a croissant on a marble table" -o hero.png

To learn more about calling multiple labs through a single gateway interface, see how a unified AI API works.

How to verify the model and per-call cost in code

The gateway runs at https://thelastapi.dev/v1/ai and is fully compatible with the standard OpenAI SDK. You can inspect the returned model and cost metadata directly in your application code:

TypeScript
import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: process.env.VIBIZ_AI_BASE_URL,
  apiKey: process.env.VIBIZ_AI_API_KEY,
});

async function runPrompt() {
  const response = await openai.chat.completions.create({
    model: 'vibiz/fast',
    messages: [{ role: 'user', content: 'Draft a three-line release note.' }],
  });

  console.log('Answer:', response.choices[0]?.message.content);

  // Inspect the gateway metadata fields returned in the JSON payload
  const rawResponse = response as typeof response & {
    served_model: string;
    cost_micro_usd: number;
  };
  console.log('Served model:', rawResponse.served_model);
  console.log('Cost in micro-USD:', rawResponse.cost_micro_usd);
}

runPrompt();

You can choose models by their stable role tier or by their raw lab identifier:

thelastapi modelUpstream lab modelGood forPrice per million tokens (in / out)
vibiz/fastGPT-6 Luna (OpenAI)Chat, tool calling, JSON mode$0.10 / $0.50
vibiz/liteDeepSeek V4.1 FlashLow-cost high-volume chat$0.14 / $0.42
vibiz/smart-miniClaude Haiku 5.5 (Anthropic)Fast writing and code tasks$0.10 / $0.50
vibiz/balancedGemini 3.8 Flash (Google)Balanced speed and quality$0.825 / $4.125
vibiz/smartClaude Sonnet 5.5 (Anthropic)Complex coding and agents$2.00 / $10.00
vibiz/proClaude Opus 5.5 (Anthropic)Long agent sessions$4.00 / $20.00

To optimize token spending across these tiers, read how to cut your LLM API bill.

How to cap spend and audit usage across projects

Beyond inspecting logs, you need hard boundaries to prevent runaways or excessive API spend. thelastapi enforces limits at three distinct levels:

  1. Daily account safety cap: Every account has a default $5 daily cap covering AI and actions combined. Account owners can raise this in Billing or via PATCH /v1/billing {"daily_spend_limit_usd": 50}. When hit, calls return a 429 status with details.reason: "daily_spend".
  2. Project AI budget: Each project defaults to a $1 daily AI budget. The owner can adjust this in the console or via PATCH /v1/projects/:id/ai {"daily_budget_usd": 10}. When hit, the gateway answers 429 with details.reason: "project_budget".
  3. End-user budget: Pass a stable pseudonymous user id via the user field in chat completions or the X-Vibiz-User HTTP header. The owner can set end_user_daily_budget_usd on the project to cap what any single end user can spend per day.

For details on configuring client caps, see per-user AI budgets.

Next: review the full list of supported chat, image, voice, and reasoning models on the AI docs page, or use vibiz ai models in your terminal to search available endpoints and their list prices.

Frequently asked questions

Do all LLM aggregators log prompt text by default?

No, aggregators differ in how they handle payloads. Some maintain persistent databases of prompt and completion text for evaluation dashboards, while others operate as stateless proxies that only record token counts and billing metadata.

How can I tell which lab model actually answered my API call?

In thelastapi, every chat completion response includes the `served_model` field in its JSON body, and audio requests return an `X-Vibiz-Served-Model` header. If a model is unavailable, the gateway returns a 503 error instead of silently routing your prompt to an alternative provider.

Where are prompts stored when generating images or audio?

When you generate media within a project, the prompt, model id, cost, and output file are saved to your project's Files with a permanent link. If you generate media using an account key outside a project, files are not saved to Files and their temporary links expire after 24 hours.

What happens if an LLM model provider experiences an outage?

thelastapi returns an HTTP 503 error with either `model_unavailable` or `provider_unavailable` alongside a retry hint. The gateway never falls back to an unrequested model behind your back.