thelastapi
Get started free

Keep every scrape: run history for web data calls

Every web data call automatically saves its input, output, duration, and cost to a searchable run history available via CLI, API, and console.

On this page 7 sections

Every web data call you make with thelastapi automatically saves its input, output, cost, and duration to run history. You do not need to configure external storage buckets, write custom database hooks, or manually archive JSON payloads. Whether you execute a scrape, crawl a domain, or pull social posts, the platform records the execution details and assigns a run identifier immediately.

This guide explains what gets stored, how to retrieve historical scraping data through the CLI and REST API, and how to prevent duplicate runs from wasting your credits.

What gets recorded for every web data call

When you call any action in the web data suite—such as web.scrape, web.crawl, or places.search—the runner captures the execution context. Every run is preserved regardless of whether it was triggered from the browser console, the command line, an autonomous agent, or your production backend.

The run history stores:

  • Action name and status: Identifies the action (such as web.scrape) and records whether it completed successfully or encountered an error.
  • Input parameters: The exact payload sent to the action, with sensitive secrets excluded and long text fields trimmed.
  • Output data: The structured JSON result up to 1 MB in size. If a massive crawl or deep page scrape exceeds this threshold, the payload is safely cut and marked with truncated: true.
  • Cost: The raw cost in micro-USD (cost_micro_usd), allowing you to trace expenses down to the exact URL.
  • Execution metadata: Total duration in milliseconds, timestamps, and error codes when calls fail.

Because run tracking is built into the action runtime, you can scrape a website to markdown with one API call without needing an extra logging database to see what was extracted yesterday.

Inspecting run history across your tools

You can inspect previous runs in four ways depending on where you are working:

InterfaceCommand or RoutePrimary Purpose
CLI summaryvibiz runs [--action <name>] [--since <window>]Review recent runs and costs from your terminal
CLI single runvibiz runs <run_id> [--json]Inspect the full input and output of a specific execution
REST API listGET /v1/actions/runsProgrammatically filter runs across projects or actions
REST API itemGET /v1/actions/runs/:idFetch the stored payload (up to 1 MB) for an application
Web Console/web-data (or the project Web data tab)Visual history, filtering, and debugging under the runner form
MCP Agent Toolvibiz_action_runsAllow AI agents to check prior results before scraping again

To view recent web actions from your local development environment, run vibiz runs:

Terminal
# List all actions run in the last 24 hours
vibiz runs --since 1d

# Filter down to scrape jobs only
vibiz runs --action web.scrape

# Scope the list to a specific project
vibiz runs --project prj_123

When you need the full output of a past extraction, pass the run identifier:

Terminal
vibiz runs arun_987654321 --json

This prints the complete stored JSON object, letting you pipe past markdown, lead lists from when you find businesses on Google Maps with an API, or social records directly into local files or analysis tools like jq.

Fetching run history through the HTTP API

Your application can retrieve past outputs without having to re-scrape websites or re-crawl directories. Every time you trigger an action over HTTP, the response includes a run_id and a direct console_url:

Terminal
curl https://thelastapi.dev/v1/actions/web.scrape \
  -H "Authorization: Bearer $VIBIZ_API_KEY" \
  -H "content-type: application/json" \
  -d '{"input":{"url":"https://example.com","mainContentOnly":true}}'

The response provides immediate metadata:

JSON
{
  "object": "action_result",
  "action": "web.scrape",
  "output": {
    "url": "https://example.com",
    "statusCode": 200,
    "title": "Example Domain",
    "markdown": "This domain is for use in illustrative examples..."
  },
  "cost_micro_usd": 800,
  "run_id": "arun_123456789",
  "console_url": "https://thelastapi.dev/web-data?action=web.scrape&run=arun_123456789"
}

Listing past runs

To view past executions programmatically, call GET /v1/actions/runs. This endpoint returns runs in reverse chronological order (newest first) without full output payloads to keep responses fast and lightweight:

Terminal
curl "https://thelastapi.dev/v1/actions/runs?action=web.scrape&limit=25" \
  -H "Authorization: Bearer $VIBIZ_API_KEY"

You can pass several optional query parameters to refine your query:

  • action: Filter by a specific action, such as web.scrape, web.crawl, or ads.library.
  • project: Pass a project id, or none to query runs not associated with any project.
  • since: Filter by an ISO 8601 timestamp.
  • cursor: A cursor string returned by previous queries for pagination.
  • limit: The number of runs to return (defaults to 25; maximum is 100).

Fetching a single run and output

Once you have a run_id, fetch the full record including its original output payload:

Terminal
curl https://thelastapi.dev/v1/actions/runs/arun_123456789 \
  -H "Authorization: Bearer $VIBIZ_API_KEY"

If the scrape originally extracted a large payload, verify whether the truncated boolean is true. Truncated records represent responses that hit the 1 MB storage limit. For crawls, full outputs contain individual page markdown blocks inside the pages array.

How AI agents reuse past runs to cut costs

Autonomous agents running loops can quickly burn through API quotas if they repeatedly scrape the same URLs. With MCP tools enabled, your agents can check existing runs before issuing new web extraction commands.

When using tools like Claude Code, Cursor, or Codex, the agent has access to vibiz_action_runs. An agent workflow typically looks like this:

  1. Check existing records: The agent calls vibiz_action_runs { action: "social.posts" } to review past scrapes when analyzing Instagram and TikTok profile and hashtag data with an API.
  2. Retrieve prior results: If the relevant target was queried recently, the agent calls vibiz_action_runs { id: "arun_..." } and consumes the cached output.
  3. Execute only when necessary: If the data is absent or outdated, the agent calls vibiz_run_action with web.scrape or web.crawl.

Additionally, the web.scrape action includes a fresh input parameter that defaults to false. When fresh is false, the service can reuse a copy fetched within the last two days rather than requesting the external web page again.

Screenshots and assets in project storage

Run history tracks textual outputs directly in the run payload, but binary assets like screenshots receive dedicated file handling.

When you run web.screenshot:

Terminal
vibiz run web.screenshot --input '{"url":"https://example.com","fullPage":true}'

The action output contains image.url. If you run the action within a project context (using a project key, passing project_id, or working within a project folder in the CLI), the screenshot image is permanently written to the project's Files storage inside the web-data folder. The resulting image.url becomes a permanent link that remains valid indefinitely.

If you execute web.screenshot without a project context, the image link expires after 24 hours. The run metadata itself remains in your history, but the binary file link will stop serving.

Web data pricing and limits

Run history does not cost extra; it is included with action execution. Every action execution reports its raw expense in cost_micro_usd. The account is billed that cost plus a 30% margin drawn from your available credits.

Action CategoryAction NameTypical Base Cost
Page Scrapesweb.scrapeFraction of a cent
Site Crawlsweb.crawl~0.1¢ per page
Web Searchesweb.searchFraction of a cent
Ad Libraryads.library~0.5¢ per call
Map Placesplaces.search~0.5¢ per call
Keyword Metricsseo.keywords~8¢ per batch (up to 1,000 keywords)

All web data actions share the account's daily spend ceiling ($5 per day by default), which can be adjusted by the account owner. If an action exceeds execution bounds (between 60 and 150 seconds depending on the action), it fails with provider_unavailable and details.reason: "timeout". Any failed action that began executing still records a run entry, logging the error and the partial cost used.

To learn more about input flags, schemas, and scraping limits, see the Web data: scrape, search, SEO, ads library documentation.

To explore run history in your own account, list your latest runs with vibiz runs or create your account at thelastapi.dev/start.

Frequently asked questions

How long are web data scrape outputs retained in run history?

Run history records are permanently retained in your console and API history. Text outputs up to 1 MB are stored with the run, while standalone screenshots generated without a project expire after 24 hours.

How can I tell if a large crawl or scrape output was cut short?

When a scrape or crawl result exceeds the 1 MB storage limit, the output payload sets the `truncated` field to `true`.

Does retrieving a past run from history cost additional credits?

No. Fetching past runs through `GET /v1/actions/runs`, `GET /v1/actions/runs/:id`, or the `vibiz runs` CLI command is free and does not charge against your daily spend ceiling.