thelastapi
Get started free

Crawl a whole site to markdown for RAG

Crawl a website to clean markdown for RAG using thelastapi web.crawl action, with path filtering and character limits in one API call.

On this page 7 sections

You can crawl an entire website into clean markdown for RAG pipelines using the web.crawl action via CLI, HTTP, or the SDK. Calling vibiz crawl <url> traverses site links, strips noise, and returns an array of pages containing URLs, titles, HTTP status codes, and clean markdown ready for chunking and embedding. One request handles discovery, extraction, and formatting without running your own browser farm.

Feeding an LLM outdated or messy HTML wastes tokens and hurts retrieval quality. When building retrieval-augmented generation (RAG) systems, turning raw pages into plain text or markdown is essential. This guide covers how web.crawl works, the inputs you can tune, and how to execute it from your terminal or application code.

Why crawl to markdown instead of raw HTML

Raw HTML contains boilerplates, scripts, style definitions, and tracking pixels that dilute context windows. Passing raw markup directly to an embedding model inflates costs and buries relevant text under technical tags.

Converting pages directly to markdown solves several retrieval problems:

  • Token efficiency. Markdown preserves structural context like headers, bullet lists, and tables without enclosing <div> tags and inline styles.
  • Hierarchical chunking. Headings (#, ##, ###) make natural boundaries when splitting long documents into semantic chunks.
  • Clean link preservation. Markdown keeps hyperlinks readable inline, letting your agent understand external references without parsing nested anchor tags.

If you only need a single documentation page or article, you can scrape a website to markdown with one API call using web.scrape. When you need entire sections, API references, or blogs ingested at once, a crawl action is the better choice.

How the web.crawl action works

The web.crawl action starts at your target URL and traverses links within the same domain. It runs synchronously, returning an array of extracted pages up to your configured limit.

Every crawl action accepts these input fields:

FieldTypeDefaultDescription
urlstringrequiredThe starting URL for the crawl.
maxPagesnumber10The number of pages to crawl (1 to 100).
includePathsarraynonePath patterns to crawl, such as ["/blog/.*"].
excludePathsarraynonePath patterns to ignore during traversal.
maxCharsPerPagenumber10000Maximum characters of markdown returned per page.

The returned JSON payload provides an array of pages, where each entry includes:

  • url: The resolved URL of the scraped page.
  • title: The <title> of the document.
  • statusCode: The HTTP status returned by the origin server.
  • markdown: The cleaned page content in markdown format.
  • truncated: A boolean flag indicating whether the content was cut off by maxCharsPerPage.

Crawls can take between 60 to 150 seconds depending on site speed and maxPages. If a crawl exceeds the internal bound, it halts and returns 503 provider_unavailable with details.reason: "timeout". To prevent timeouts on slower hosts, set a lower maxPages value.

Crawling a site from the terminal

You can run crawls directly with the vibiz CLI. For a quick crawl using default limits:

Terminal
vibiz crawl https://example.com --max-pages 25

When you need granular control over paths, use vibiz run with explicit JSON input:

Terminal
vibiz run web.crawl --input '{
  "url": "https://example.com",
  "maxPages": 20,
  "includePaths": ["/docs/.*"],
  "excludePaths": ["/docs/deprecated/.*"],
  "maxCharsPerPage": 15000
}'

Every command prints the status, the markdown output, and the exact cost in micro-USD (cost_micro_usd).

Running crawls in code and agents

You can run crawls inside your applications using the @vibiz/app SDK or standard HTTP requests against the web data endpoints.

Using the SDK

Inside a project using the @vibiz/app client, call vibiz.actions.run:

TypeScript
import { vibiz } from '@vibiz/app';

interface CrawledPage {
  url: string;
  title: string;
  statusCode: number;
  markdown: string;
  truncated: boolean;
}

const run = await vibiz.actions.run('web.crawl', {
  url: 'https://example.com',
  maxPages: 15,
  includePaths: ['/guides/.*'],
  maxCharsPerPage: 8000,
});

const pages: CrawledPage[] = run.output.pages;

for (const page of pages) {
  console.log(`Indexed: ${page.title} (${page.url})`);
  // Pass page.markdown to your text splitter and embedding model
}

Using HTTP

To trigger a crawl from Python, Go, or any other language, send a POST request to /v1/actions/web.crawl:

Terminal
curl https://thelastapi.dev/v1/actions/web.crawl \
  -H "Authorization: Bearer $VIBIZ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "url": "https://example.com",
      "maxPages": 10,
      "includePaths": ["/blog/.*"]
    }
  }'

The response contains the run ID, execution cost, and output:

JSON
{
  "object": "action_result",
  "action": "web.crawl",
  "output": {
    "pages": [
      {
        "url": "https://example.com/blog/getting-started",
        "title": "Getting Started",
        "statusCode": 200,
        "markdown": "# Getting Started\n\nWelcome to the platform...",
        "truncated": false
      }
    ]
  },
  "cost_micro_usd": 10000,
  "run_id": "arun_123",
  "console_url": "https://thelastapi.dev/web-data?action=web.crawl&run=arun_123"
}

If you are using AI agents like Cursor, Claude Code, or Codex via MCP, the agent can call vibiz_run_action with web.crawl directly to gather fresh web context.

Managing history and inspecting past runs

All crawl runs are recorded in your account history, regardless of whether they originated from the CLI, the API, the console, or an agent.

You can inspect previous runs from the terminal:

Terminal
vibiz runs --action web.crawl --since 1d
vibiz runs arun_123

To view past operations in the browser, visit the Web data dashboard in the console. The console lets you filter runs by action type, view the resulting payload (up to 1 MB), and inspect the exact input parameters used.

Agents can also check past runs using the MCP tool vibiz_action_runs to avoid re-crawling the same site and wasting credits.

Pricing and spend limits

Web data actions charge what the underlying extraction costs (cost_micro_usd) plus a 30% margin, which is drawn from your account credits.

  • A crawl costs about a tenth of a cent per page ($0.001/page).
  • A 10-page crawl typically costs around 1 cent ($0.01).
  • Every call counts against your account's daily spend ceiling ($5 default, configurable by the owner in Billing).
  • If your account runs out of credits, the API returns a 402 payment_required status.

If you also need visual validation of the pages you are ingesting, you can screenshot a web page with an API using web.screenshot to review layout rendering alongside text extraction. For monitoring when external documentation sources shift or vanish, see our guide on how to know when your site goes down, a deploy fails or sales stop.

To start crawling documentation, test vibiz crawl <url> using the CLI or inspect the parameter schemas directly in the Web data documentation.

Frequently asked questions

How many pages can I crawl in a single request?

You can set `maxPages` to any value between 1 and 100, with a default limit of 10 pages per call.

How does path filtering work in web.crawl?

The action accepts `includePaths` and `excludePaths` as regex patterns (such as `["/blog/.*"]`) to restrict the crawl to specific URL sections.

What happens if a crawl takes too long?

Crawl requests that exceed internal processing bounds (between 60 and 150 seconds) fail with a `503 provider_unavailable` error and `details.reason: "timeout"`.

Where can I review the output of an earlier crawl?

You can retrieve past runs using the CLI with `vibiz runs <run_id>`, through the API at `GET /v1/actions/runs/:id`, or under Web data in the console.