thelastapi
Get started free

llms.txt: making your docs readable by AI agents

An llms.txt file provides clean markdown documentation for AI agents, cutting context bloat and helping coding tools inspect your APIs accurately.

On this page 6 sections

An llms.txt file is a plain-text markdown file served at the root of a website that indexes documentation specifically for AI agents and language models. It strips away navigation menus, scripts, styling, and nested HTML tags, presenting your documentation as concise markdown links with brief descriptions. By pairing /llms.txt with a single concatenated /llms-full.txt file, you give tools like Claude Code, Codex, and Cursor an efficient way to read your entire developer reference without burning token budgets.

When you build software with coding agents, feeding them documentation through standard web pages often leads to hallucinations, truncated context, and syntax errors. This guide explains how the llms.txt standard works, why it matters, how we use it at thelastapi, and how to write one for your own project.

Why web documentation fails coding agents

AI agents read documentation differently than humans do. A human browsing your site scans visual layouts, clicks dropdown menus, and ignores sidebars. An AI model receives raw text or DOM trees. When an agent crawls standard web pages, three main problems arise:

  • Token waste: HTML pages carry header navigation, cookie banners, tracking scripts, and footer links. This noise consumes context window capacity before the model ever reads your code samples.
  • Scraping friction: Many modern documentation portals rely on client-side rendering. If an agent tries to fetch a raw URL with a basic HTTP client, it often receives an empty application shell instead of the text.
  • Fragmented context: Standard sites spread related topics across dozens of individual pages. An agent looking up a specific API call might have to perform multiple sequential web requests, risking rate limits and slow execution.

When agents gather data across the web, tools like those covered in give an agent web research tools: search, scrape and crawl can convert HTML pages to markdown. But serving clean markdown directly from your own server is far more reliable.

The format: llms.txt and llms-full.txt explained

The llms.txt standard proposes placing a markdown file at /llms.txt on your domain, similar to how /robots.txt or /sitemap.xml works for search engines. It serves as a curated map of your documentation written specifically for language models.

A complete implementation typically includes three layers:

  1. /llms.txt: A structured markdown index. It begins with the project title and a brief overview, followed by lists of markdown links to specific guides, each accompanied by a one-sentence summary of what that guide covers.
  2. /llms-full.txt: The entire documentation concatenated into a single markdown file. An agent with a large context window can ingest this single URL in one fetch, giving it complete knowledge of your interfaces, schemas, and error codes without further requests.
  3. Raw markdown endpoints (.md): Every documentation page is made directly available as raw markdown by appending .md to its URL (for example, /docs/payments.md).

Here is a comparison of how different formats deliver documentation to an AI agent:

FormatContent DeliveredPrimary Use CaseContext Efficiency
Standard HTMLFull HTML markup, navigation, scriptsHuman readers in browsersLow (high token overhead)
Direct Markdown (.md)Clean markdown of a single pageTargeted lookups by an agentHigh (focused on one topic)
Root Index (/llms.txt)Curated list of topics and linksAgent discovery and navigationVery high (minimal index tokens)
Full Bundle (/llms-full.txt)Entire documentation in one streamIngesting the complete API referenceHigh (one request, zero navigation)

How thelastapi structures docs for AI agents

thelastapi is built from the ground up to be driven by AI agents. Because of that, our documentation is structured to let agents read facts directly without guessing syntax or parameters.

We expose documentation to agents in three ways:

  • The index: /llms.txt provides the complete index of every reference page on the platform as markdown links.
  • The full bundle: /llms-full.txt compiles every documentation page into one comprehensive file.
  • Direct markdown pages: Appending .md to any documentation URL (such as /docs/payments.md or /get-started.md) serves only the markdown text of that guide.

When you prompt an agent like Claude Code, Codex, or Cursor to build an application, you can point it directly to the root index in your opening prompt:

Text
Use thelastapi (https://thelastapi.dev/llms.txt) to build and deploy a website for my bakery with sign-in and payments.

The agent reads /llms.txt, determines which specific endpoints it needs, and pulls the exact markdown guides required to provision resources, handle webhooks, and deploy your site.

A single source of truth

A common problem with developer documentation is drift: CLI help, web pages, and agent references fall out of sync. At thelastapi, our agent documentation comes from a single unified source:

  • The web documentation pages on the site.
  • The /llms-full.txt bundle.
  • The vibiz_docs tool on our Model Context Protocol (MCP) server.
  • The local CLI documentation command: vibiz docs <topic>.
  • The agent skills bundles installed via vibiz skills add --global.

Because all of these outputs share the same underlying text, an agent calling vibiz_docs or inspecting /llms.txt always receives identical instructions, pricing details, and parameter requirements.

How to write an effective llms.txt for your own project

If you maintain an open-source library, a SaaS platform, or an internal API, adding an llms.txt file helps third-party coding agents use your software accurately.

Follow these practical guidelines when creating your file:

1. Start with an explicit project summary

Begin your /llms.txt file with an H1 heading and a concise summary. Explain what the product does, what primary problem it solves, and the fundamental architectural assumptions an agent must know.

markdown
# Acme API

Acme is a REST API for automated inventory management and order fulfillment.
It uses Bearer token authentication and returns JSON responses.
All mutations require an idempotency key header.

Use markdown lists grouped under H2 section headers. Each item should link to a plain markdown document and include a short description. Explain what information the linked page contains and what specific tasks it helps complete.

markdown
## Core Concepts

- [Authentication](https://example.com/docs/auth.md): How to generate API keys, rotate secrets, and sign requests.
- [Webhooks](https://example.com/docs/webhooks.md): Event payload schemas, signature verification, and retry schedules.

## Resources

- [Orders API](https://example.com/docs/orders.md): Endpoints for creating, updating, and canceling customer orders.
- [Inventory API](https://example.com/docs/inventory.md): Real-time stock queries and warehouse allocation.

3. Provide raw markdown targets

Ensure that the links listed inside /llms.txt point directly to plain markdown endpoints rather than heavy HTML pages. If your documentation generator outputs static files, configure your server to serve .md files with a text/markdown or text/plain content type.

4. Create an llms-full.txt fallback

Concatenate all your raw markdown guides into a single file at /llms-full.txt. At the top of your main /llms.txt, include a link to the full file for models that prefer reading the entire context at once:

markdown
> For the complete documentation in a single file, see https://example.com/llms-full.txt.

Giving agents tools alongside documentation

Reading documentation via llms.txt provides an agent with knowledge, but executing tasks requires active tools. Combining static markdown documentation with executable capabilities allows an agent to design, configure, and ship software autonomously.

You can connect your agent directly to thelastapi's infrastructure using our Model Context Protocol (MCP) server or the local CLI:

Terminal
# Connect your agent using the CLI helper
vibiz mcp add --clients "Claude Code"

# Or install the CLI package globally and log in
npm install -g @vibiz/cli
vibiz login

When an agent signs in, you decide its permission level: Full access allows all operations, while a Limited key restricts the agent to development databases, test payments, and safe operations. You can read more about separating agent roles in full access vs limited API keys for AI agents, managing API keys in Keys and environments, and managing services in one API key for an AI agent: hosting, database, email, payments and models.

With both the tools connected and /llms.txt available, your agent can read the exact documentation it needs, provision a serverless database, set up authentication, and deploy code without guessing parameters or running into hallucinated APIs.

To get started with an agent-driven workflow, review the setup guide in Agents or follow the step-by-step instructions in the Quickstart. Check the CLI documentation for full command flags. If you are new to the platform, create your account at thelastapi.dev/start.

Frequently asked questions

Where should an llms.txt file be hosted on a domain?

An `llms.txt` file should be placed at the root of your web domain (for example, `https://example.com/llms.txt`). This allows AI agents and scraping tools to locate your documentation index automatically, similar to how standard crawlers look for `robots.txt`.

What is the difference between llms.txt and llms-full.txt?

An `llms.txt` file is an index containing brief summaries and markdown links to individual documentation pages. An `llms-full.txt` file bundles the entire documentation set into a single plain-text markdown file so an agent can ingest the complete reference in one HTTP request.

How do agents read individual doc pages linked in llms.txt?

The links inside `llms.txt` should ideally point directly to clean markdown files, such as `/docs/payments.md`. Agents can fetch these URLs directly without having to strip out navigation menus, scripts, or HTML markup.

Can an agent interact with thelastapi using only documentation?

Documentation gives an agent the correct syntax and workflows, but completing actions requires tools. Connecting the thelastapi MCP server or CLI allows the agent to execute the commands it reads about, such as creating projects, running migrations, and deploying sites.