Scrape a website to markdown with one API call
Scrape any web page to clean markdown using the vibiz CLI or a single POST call to /v1/actions/web.scrape with your unified thelastapi key.
On this page 7 sections
To scrape a website to markdown with one API call, send a POST request to /v1/actions/web.scrape with the target URL, or run vibiz scrape <url> from your terminal. You receive clean markdown stripped of navigation and boilerplate, alongside the page title, description, language, and HTTP status code. The call takes care of page rendering, costs a fraction of a cent, and runs on the single API key you use across thelastapi.
Turning web pages into markdown is essential when feeding live content to LLMs, generating summaries, or storing external documentation. Instead of running a headless browser farm, rotating proxies, and writing HTML-to-markdown parsers, you make one call and receive structured text.
How to scrape a page from the CLI or HTTP
You can test scraping directly in your terminal using the CLI, or send an HTTP request from any backend language.
From the command line:
vibiz scrape https://example.comTo see the complete structured output as JSON, pass the --json flag:
vibiz scrape https://example.com --jsonWith plain HTTP, make a POST request to the web data endpoint:
curl https://thelastapi.dev/v1/actions/web.scrape \
-H "Authorization: Bearer $VIBIZ_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input":{"url":"https://example.com"}}'Every response returns the parsed content, metadata, the exact cost in micro-USD, and a link to view the run in the console:
{
"object": "action_result",
"action": "web.scrape",
"output": {
"url": "https://example.com",
"statusCode": 200,
"title": "Example Domain",
"description": "This domain is for use in illustrative examples in documents.",
"language": "en",
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"truncated": false
},
"cost_micro_usd": 1500,
"run_id": "arun_9f82c1b7",
"console_url": "https://thelastapi.dev/web-data?action=web.scrape&run=arun_9f82c1b7"
}The scraper isolates the core article or documentation body by default, removing headers, footers, cookie banners, and nav bars so your prompt context is not wasted on navigation links.
Fine-tuning the markdown output
The web.scrape action accepts several optional parameters to control how much text is returned, whether links are kept, and how aggressively the page should be cached.
| Field | Type | Default | Description |
|---|---|---|---|
url | string (required) | — | The full web URL to fetch and convert. |
mainContentOnly | boolean | true | When true, strips headers, footers, sidebars, and nav links. Set to false to capture the entire raw page body. |
includeLinks | boolean | false | When true, returns an array of extracted hyperlinks found on the page in output.links. |
maxChars | number | 20000 | Maximum character length for markdown output (allowed range: 500 to 100000). |
fresh | boolean | false | When false, answers from a cached copy up to 2 days old if available. Set to true to force a new fetch. |
Here is an example requesting links and capping output at 5,000 characters:
curl https://thelastapi.dev/v1/actions/web.scrape \
-H "Authorization: Bearer $VIBIZ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"url": "https://example.com",
"includeLinks": true,
"maxChars": 5000,
"fresh": true
}
}'If the extracted content exceeds maxChars, the response sets "truncated": true, letting your application know that additional content was trimmed.
Scraping from Node.js and TypeScript apps
In an application built on thelastapi, you use @vibiz/app to call actions directly. The SDK automatically picks up your project credentials:
import { vibiz } from '@vibiz/app';
interface ScrapeResult {
url: string;
statusCode: number;
title: string;
description: string;
language: string;
markdown: string;
truncated: boolean;
links?: string[];
}
const run = await vibiz.actions.run('web.scrape', {
url: 'https://example.com',
mainContentOnly: true,
maxChars: 15000,
});
const page = run.output as ScrapeResult;
console.log(`Scraped: ${page.title} (${page.markdown.length} characters)`);If you are feeding scraped markdown straight to an AI model, you can use the unified AI gateway on the exact same API key. For instance, you can pipe the scraped markdown into Moonshot AI models or Anthropic models using the standard OpenAI client format; see our guide on Kimi API access: call Moonshot AI models with an OpenAI client.
For developers using AI coding agents like Cursor, Claude Code, or Codex, thelastapi exposes MCP tools. An agent can call vibiz_run_action with name: "web.scrape" to read documentation pages or live articles directly into its workspace.
The wider web data and SEO suite
Scraping a single page is often only the first step in research, SEO tracking, or AI workflows. thelastapi provides a complete family of synchronous web data actions documented in Web data: scrape, search, SEO, ads library.
- Multi-page site crawls: When you need an entire site converted to markdown rather than a single URL, use
web.crawl. It traverses up to 100 pages with path filters; read Crawl a whole site to markdown for RAG to learn how to structure site-wide ingestion. - Visual page captures: To store a visual archive alongside your text,
web.screenshotcaptures desktop or mobile PNGs and saves them directly to your project files. Learn more in Screenshot a web page with an API. - Design extraction: To inspect how another site or your own site is styled,
web.designanalyzes typography scales, colors, spacing rhythm, and radii. See Design: learn a site's design system for details on exporting Tailwind v4 and CSS tokens. - Search and SERP data: Run web search queries using
web.search(vibiz search "<query>"), read live Google rankings withseo.serp, or generate keyword search volumes and difficulty scores withseo.keyword_ideas. See our walkthrough on Keyword ideas with volume and difficulty from one call. - Public business and social data: Pull active advertisements from the Meta Ad Library (
ads.library), search local listings on Google Maps (places.search), fetch customer reviews from Google or Trustpilot (reviews), read public profiles from TikTok and Instagram (social.profile), or pull corporate profiles withlinkedin.company.
Every action follows the same synchronous input and output pattern, returns cost_micro_usd, and is billed to your unified balance.
Run history, costs, and error handling
Every scrape or action call is stored automatically. You can review past runs in the console under Web data, or view them through the CLI:
# List recent runs across all actions
vibiz runs --since 1d
# Filter to web.scrape runs only
vibiz runs --action web.scrape
# Inspect a specific run by ID
vibiz runs arun_9f82c1b7You can also fetch past runs over HTTP via GET /v1/actions/runs or inspect a single run at GET /v1/actions/runs/:id.
Pricing and budgets
Scraping a page costs a fraction of a cent. Each response returns cost_micro_usd, which represents the underlying action cost. The account is billed that amount plus a 30% margin on web data, deducted from your pre-funded credit balance.
All actions count toward your account's daily spend ceiling ($5 per day by default, configurable by the account owner in Billing). If a call runs up against that limit, the API answers with status code 429 and code rate_limited.
Error codes
When something goes wrong, the API answers with structured JSON error details:
400 invalid_request: The input URL is invalid or malformed. Inspectdetails.fieldto identify the issue.402 payment_required: Your account credit balance is exhausted. Top up credits viavibiz billing top-upor in the console.404 not_found: The specified action does not exist, or the target website responded with a 404 HTTP status.429 rate_limited: You have hit the account daily spend ceiling.503 provider_unavailable: The request timed out while loading the remote website. Try again, or ensure the destination site is reachable.
Next steps
Start by scraping a page from your terminal with vibiz scrape <url>, or integrate POST /v1/actions/web.scrape into your application. If you need to index multiple pages for an AI knowledge base, explore our tutorial on crawling a whole site to markdown for RAG.
Frequently asked questions
Does the scrape endpoint render JavaScript on client-heavy sites?
Yes. The scraper loads the target URL in a headless browser environment to ensure client-rendered elements, dynamically inserted text, and lazy-loaded assets are captured before producing the markdown.
How does the scraper isolate the main content of a page?
By default, `mainContentOnly` is set to `true`, which strips away common boilerplate like navigation bars, headers, footers, and cookie popups. If you want the full page body instead, pass `"mainContentOnly": false` in your request input.
Where can I see past scrape runs and their output?
You can inspect every past run in the console under Web data, list them from the CLI with `vibiz runs --action web.scrape`, or fetch them via `GET /v1/actions/runs`. Outputs up to 1 MB are retained.
How much does scraping a page cost?
Scraping a page costs a fraction of a cent. Each API response includes the exact cost in `cost_micro_usd`, and the account is billed that cost plus a 30% margin drawn from your credit balance.