# Scrape a website to markdown with one API call

> Scrape any web page to clean markdown using the vibiz CLI or a single POST call to /v1/actions/web.scrape with your unified thelastapi key.

**To scrape a website to markdown with one API call, send a `POST` request to `/v1/actions/web.scrape` with the target URL, or run `vibiz scrape <url>` from your terminal.** You receive clean markdown stripped of navigation and boilerplate, alongside the page title, description, language, and HTTP status code. The call takes care of page rendering, costs a fraction of a cent, and runs on the single API key you use across thelastapi.

Turning web pages into markdown is essential when feeding live content to LLMs, generating summaries, or storing external documentation. Instead of running a headless browser farm, rotating proxies, and writing HTML-to-markdown parsers, you make one call and receive structured text.

## How to scrape a page from the CLI or HTTP

You can test scraping directly in your terminal using the CLI, or send an HTTP request from any backend language.

From the command line:

```bash
vibiz scrape https://example.com
```

To see the complete structured output as JSON, pass the `--json` flag:

```bash
vibiz scrape https://example.com --json
```

With plain HTTP, make a `POST` request to the web data endpoint:

```bash
curl https://thelastapi.dev/v1/actions/web.scrape \
  -H "Authorization: Bearer $VIBIZ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input":{"url":"https://example.com"}}'
```

Every response returns the parsed content, metadata, the exact cost in micro-USD, and a link to view the run in the console:

```json
{
  "object": "action_result",
  "action": "web.scrape",
  "output": {
    "url": "https://example.com",
    "statusCode": 200,
    "title": "Example Domain",
    "description": "This domain is for use in illustrative examples in documents.",
    "language": "en",
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "truncated": false
  },
  "cost_micro_usd": 1500,
  "run_id": "arun_9f82c1b7",
  "console_url": "https://thelastapi.dev/web-data?action=web.scrape&run=arun_9f82c1b7"
}
```

The scraper isolates the core article or documentation body by default, removing headers, footers, cookie banners, and nav bars so your prompt context is not wasted on navigation links.

## Fine-tuning the markdown output

The `web.scrape` action accepts several optional parameters to control how much text is returned, whether links are kept, and how aggressively the page should be cached.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string (required) | — | The full web URL to fetch and convert. |
| `mainContentOnly` | boolean | `true` | When true, strips headers, footers, sidebars, and nav links. Set to `false` to capture the entire raw page body. |
| `includeLinks` | boolean | `false` | When true, returns an array of extracted hyperlinks found on the page in `output.links`. |
| `maxChars` | number | `20000` | Maximum character length for markdown output (allowed range: `500` to `100000`). |
| `fresh` | boolean | `false` | When false, answers from a cached copy up to 2 days old if available. Set to `true` to force a new fetch. |

Here is an example requesting links and capping output at 5,000 characters:

```bash
curl https://thelastapi.dev/v1/actions/web.scrape \
  -H "Authorization: Bearer $VIBIZ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "url": "https://example.com",
      "includeLinks": true,
      "maxChars": 5000,
      "fresh": true
    }
  }'
```

If the extracted content exceeds `maxChars`, the response sets `"truncated": true`, letting your application know that additional content was trimmed.

## Scraping from Node.js and TypeScript apps

In an application built on thelastapi, you use `@vibiz/app` to call actions directly. The SDK automatically picks up your project credentials:

```ts
import { vibiz } from '@vibiz/app';

interface ScrapeResult {
  url: string;
  statusCode: number;
  title: string;
  description: string;
  language: string;
  markdown: string;
  truncated: boolean;
  links?: string[];
}

const run = await vibiz.actions.run('web.scrape', {
  url: 'https://example.com',
  mainContentOnly: true,
  maxChars: 15000,
});

const page = run.output as ScrapeResult;
console.log(`Scraped: ${page.title} (${page.markdown.length} characters)`);
```

If you are feeding scraped markdown straight to an AI model, you can use the unified AI gateway on the exact same API key. For instance, you can pipe the scraped markdown into Moonshot AI models or Anthropic models using the standard OpenAI client format; see our guide on [Kimi API access: call Moonshot AI models with an OpenAI client](https://thelastapi.dev/blog/kimi-api-access).

For developers using AI coding agents like Cursor, Claude Code, or Codex, thelastapi exposes MCP tools. An agent can call `vibiz_run_action` with `name: "web.scrape"` to read documentation pages or live articles directly into its workspace.

## The wider web data and SEO suite

Scraping a single page is often only the first step in research, SEO tracking, or AI workflows. thelastapi provides a complete family of synchronous web data actions documented in [Web data: scrape, search, SEO, ads library](https://thelastapi.dev/docs/actions).

- **Multi-page site crawls:** When you need an entire site converted to markdown rather than a single URL, use `web.crawl`. It traverses up to 100 pages with path filters; read [Crawl a whole site to markdown for RAG](https://thelastapi.dev/blog/crawl-site-markdown-rag) to learn how to structure site-wide ingestion.
- **Visual page captures:** To store a visual archive alongside your text, `web.screenshot` captures desktop or mobile PNGs and saves them directly to your project files. Learn more in [Screenshot a web page with an API](https://thelastapi.dev/blog/screenshot-webpage-api).
- **Design extraction:** To inspect how another site or your own site is styled, `web.design` analyzes typography scales, colors, spacing rhythm, and radii. See [Design: learn a site's design system](https://thelastapi.dev/docs/design) for details on exporting Tailwind v4 and CSS tokens.
- **Search and SERP data:** Run web search queries using `web.search` (`vibiz search "<query>"`), read live Google rankings with `seo.serp`, or generate keyword search volumes and difficulty scores with `seo.keyword_ideas`. See our walkthrough on [Keyword ideas with volume and difficulty from one call](https://thelastapi.dev/blog/keyword-ideas-api).
- **Public business and social data:** Pull active advertisements from the Meta Ad Library (`ads.library`), search local listings on Google Maps (`places.search`), fetch customer reviews from Google or Trustpilot (`reviews`), read public profiles from TikTok and Instagram (`social.profile`), or pull corporate profiles with `linkedin.company`.

Every action follows the same synchronous input and output pattern, returns `cost_micro_usd`, and is billed to your unified balance.

## Run history, costs, and error handling

Every scrape or action call is stored automatically. You can review past runs in the console under **Web data**, or view them through the CLI:

```bash
# List recent runs across all actions
vibiz runs --since 1d

# Filter to web.scrape runs only
vibiz runs --action web.scrape

# Inspect a specific run by ID
vibiz runs arun_9f82c1b7
```

You can also fetch past runs over HTTP via `GET /v1/actions/runs` or inspect a single run at `GET /v1/actions/runs/:id`.

### Pricing and budgets

Scraping a page costs a fraction of a cent. Each response returns `cost_micro_usd`, which represents the underlying action cost. The account is billed that amount plus a 30% margin on web data, deducted from your pre-funded credit balance. 

All actions count toward your account's daily spend ceiling ($5 per day by default, configurable by the account owner in Billing). If a call runs up against that limit, the API answers with status code `429` and code `rate_limited`.

### Error codes

When something goes wrong, the API answers with structured JSON error details:

- **`400 invalid_request`**: The input URL is invalid or malformed. Inspect `details.field` to identify the issue.
- **`402 payment_required`**: Your account credit balance is exhausted. Top up credits via `vibiz billing top-up` or in the console.
- **`404 not_found`**: The specified action does not exist, or the target website responded with a 404 HTTP status.
- **`429 rate_limited`**: You have hit the account daily spend ceiling.
- **`503 provider_unavailable`**: The request timed out while loading the remote website. Try again, or ensure the destination site is reachable.

## Next steps

Start by scraping a page from your terminal with `vibiz scrape <url>`, or integrate `POST /v1/actions/web.scrape` into your application. If you need to index multiple pages for an AI knowledge base, explore our tutorial on [crawling a whole site to markdown for RAG](https://thelastapi.dev/blog/crawl-site-markdown-rag).

## Frequently asked questions

### Does the scrape endpoint render JavaScript on client-heavy sites?

Yes. The scraper loads the target URL in a headless browser environment to ensure client-rendered elements, dynamically inserted text, and lazy-loaded assets are captured before producing the markdown.

### How does the scraper isolate the main content of a page?

By default, `mainContentOnly` is set to `true`, which strips away common boilerplate like navigation bars, headers, footers, and cookie popups. If you want the full page body instead, pass `"mainContentOnly": false` in your request input.

### Where can I see past scrape runs and their output?

You can inspect every past run in the console under Web data, list them from the CLI with `vibiz runs --action web.scrape`, or fetch them via `GET /v1/actions/runs`. Outputs up to 1 MB are retained.

### How much does scraping a page cost?

Scraping a page costs a fraction of a cent. Each API response includes the exact cost in `cost_micro_usd`, and the account is billed that cost plus a 30% margin drawn from your credit balance.

---

Published 2026-11-15 · https://thelastapi.dev/blog/scrape-website-to-markdown-api
