# Crawl a whole site to markdown for RAG

> Crawl a website to clean markdown for RAG using thelastapi web.crawl action, with path filtering and character limits in one API call.

**You can crawl an entire website into clean markdown for RAG pipelines using the `web.crawl` action via CLI, HTTP, or the SDK.** Calling `vibiz crawl <url>` traverses site links, strips noise, and returns an array of pages containing URLs, titles, HTTP status codes, and clean markdown ready for chunking and embedding. One request handles discovery, extraction, and formatting without running your own browser farm.

Feeding an LLM outdated or messy HTML wastes tokens and hurts retrieval quality. When building retrieval-augmented generation (RAG) systems, turning raw pages into plain text or markdown is essential. This guide covers how `web.crawl` works, the inputs you can tune, and how to execute it from your terminal or application code.

## Why crawl to markdown instead of raw HTML

Raw HTML contains boilerplates, scripts, style definitions, and tracking pixels that dilute context windows. Passing raw markup directly to an embedding model inflates costs and buries relevant text under technical tags.

Converting pages directly to markdown solves several retrieval problems:

- **Token efficiency.** Markdown preserves structural context like headers, bullet lists, and tables without enclosing `<div>` tags and inline styles.
- **Hierarchical chunking.** Headings (`#`, `##`, `###`) make natural boundaries when splitting long documents into semantic chunks.
- **Clean link preservation.** Markdown keeps hyperlinks readable inline, letting your agent understand external references without parsing nested anchor tags.

If you only need a single documentation page or article, you can [scrape a website to markdown with one API call](https://thelastapi.dev/blog/scrape-website-to-markdown-api) using `web.scrape`. When you need entire sections, API references, or blogs ingested at once, a crawl action is the better choice.

## How the web.crawl action works

The `web.crawl` action starts at your target URL and traverses links within the same domain. It runs synchronously, returning an array of extracted pages up to your configured limit.

Every crawl action accepts these input fields:

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string | *required* | The starting URL for the crawl. |
| `maxPages` | number | `10` | The number of pages to crawl (1 to 100). |
| `includePaths` | array | *none* | Path patterns to crawl, such as `["/blog/.*"]`. |
| `excludePaths` | array | *none* | Path patterns to ignore during traversal. |
| `maxCharsPerPage` | number | `10000` | Maximum characters of markdown returned per page. |

The returned JSON payload provides an array of `pages`, where each entry includes:
- `url`: The resolved URL of the scraped page.
- `title`: The `<title>` of the document.
- `statusCode`: The HTTP status returned by the origin server.
- `markdown`: The cleaned page content in markdown format.
- `truncated`: A boolean flag indicating whether the content was cut off by `maxCharsPerPage`.

Crawls can take between 60 to 150 seconds depending on site speed and `maxPages`. If a crawl exceeds the internal bound, it halts and returns `503 provider_unavailable` with `details.reason: "timeout"`. To prevent timeouts on slower hosts, set a lower `maxPages` value.

## Crawling a site from the terminal

You can run crawls directly with the `vibiz` CLI. For a quick crawl using default limits:

```bash
vibiz crawl https://example.com --max-pages 25
```

When you need granular control over paths, use `vibiz run` with explicit JSON input:

```bash
vibiz run web.crawl --input '{
  "url": "https://example.com",
  "maxPages": 20,
  "includePaths": ["/docs/.*"],
  "excludePaths": ["/docs/deprecated/.*"],
  "maxCharsPerPage": 15000
}'
```

Every command prints the status, the markdown output, and the exact cost in micro-USD (`cost_micro_usd`).

## Running crawls in code and agents

You can run crawls inside your applications using the `@vibiz/app` SDK or standard HTTP requests against the web data endpoints.

### Using the SDK

Inside a project using the `@vibiz/app` client, call `vibiz.actions.run`:

```ts
import { vibiz } from '@vibiz/app';

interface CrawledPage {
  url: string;
  title: string;
  statusCode: number;
  markdown: string;
  truncated: boolean;
}

const run = await vibiz.actions.run('web.crawl', {
  url: 'https://example.com',
  maxPages: 15,
  includePaths: ['/guides/.*'],
  maxCharsPerPage: 8000,
});

const pages: CrawledPage[] = run.output.pages;

for (const page of pages) {
  console.log(`Indexed: ${page.title} (${page.url})`);
  // Pass page.markdown to your text splitter and embedding model
}
```

### Using HTTP

To trigger a crawl from Python, Go, or any other language, send a `POST` request to `/v1/actions/web.crawl`:

```bash
curl https://thelastapi.dev/v1/actions/web.crawl \
  -H "Authorization: Bearer $VIBIZ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "url": "https://example.com",
      "maxPages": 10,
      "includePaths": ["/blog/.*"]
    }
  }'
```

The response contains the run ID, execution cost, and output:

```json
{
  "object": "action_result",
  "action": "web.crawl",
  "output": {
    "pages": [
      {
        "url": "https://example.com/blog/getting-started",
        "title": "Getting Started",
        "statusCode": 200,
        "markdown": "# Getting Started\n\nWelcome to the platform...",
        "truncated": false
      }
    ]
  },
  "cost_micro_usd": 10000,
  "run_id": "arun_123",
  "console_url": "https://thelastapi.dev/web-data?action=web.crawl&run=arun_123"
}
```

If you are using AI agents like Cursor, Claude Code, or Codex via MCP, the agent can call `vibiz_run_action` with `web.crawl` directly to gather fresh web context.

## Managing history and inspecting past runs

All crawl runs are recorded in your account history, regardless of whether they originated from the CLI, the API, the console, or an agent.

You can inspect previous runs from the terminal:

```bash
vibiz runs --action web.crawl --since 1d
vibiz runs arun_123
```

To view past operations in the browser, visit the [Web data](https://thelastapi.dev/docs/actions) dashboard in the console. The console lets you filter runs by action type, view the resulting payload (up to 1 MB), and inspect the exact input parameters used.

Agents can also check past runs using the MCP tool `vibiz_action_runs` to avoid re-crawling the same site and wasting credits.

## Pricing and spend limits

Web data actions charge what the underlying extraction costs (`cost_micro_usd`) plus a 30% margin, which is drawn from your account credits.

- A crawl costs about a tenth of a cent per page ($0.001/page).
- A 10-page crawl typically costs around 1 cent ($0.01).
- Every call counts against your account's daily spend ceiling ($5 default, configurable by the owner in Billing).
- If your account runs out of credits, the API returns a `402 payment_required` status.

If you also need visual validation of the pages you are ingesting, you can [screenshot a web page with an API](https://thelastapi.dev/blog/screenshot-webpage-api) using `web.screenshot` to review layout rendering alongside text extraction. For monitoring when external documentation sources shift or vanish, see our guide on how to [know when your site goes down, a deploy fails or sales stop](https://thelastapi.dev/blog/know-when-your-site-goes-down).

To start crawling documentation, test `vibiz crawl <url>` using the CLI or inspect the parameter schemas directly in the [Web data](https://thelastapi.dev/docs/actions) documentation.

## Frequently asked questions

### How many pages can I crawl in a single request?

You can set `maxPages` to any value between 1 and 100, with a default limit of 10 pages per call.

### How does path filtering work in web.crawl?

The action accepts `includePaths` and `excludePaths` as regex patterns (such as `["/blog/.*"]`) to restrict the crawl to specific URL sections.

### What happens if a crawl takes too long?

Crawl requests that exceed internal processing bounds (between 60 and 150 seconds) fail with a `503 provider_unavailable` error and `details.reason: "timeout"`.

### Where can I review the output of an earlier crawl?

You can retrieve past runs using the CLI with `vibiz runs <run_id>`, through the API at `GET /v1/actions/runs/:id`, or under Web data in the console.

---

Published 2026-11-15 · https://thelastapi.dev/blog/crawl-site-markdown-rag
