thelastapi
Get started free

Give an agent web research tools: search, scrape and crawl

Give your AI agent web search, page scraping, and site crawling with one key using thelastapi via the CLI, Model Context Protocol, or HTTP.

On this page 8 sections

You can give an AI agent web search, page scraping, and multi-page crawling through one key using thelastapi's built-in actions and Model Context Protocol (MCP) server. Instead of setting up separate search APIs, headless browser scrapers, and crawling fleets, your agent can search query results, turn web pages into clean markdown, and crawl whole domains using unified CLI commands or MCP tools. Every run is recorded in your project history, billed against your account credits, and capped by your daily spend limit.

This guide shows how web research tools work for agents, how to invoke them from terminal workflows or MCP clients, and how to monitor past runs.

Why web research needs dedicated tools for agents

Language models make mistakes when guessing live URLs, outdated facts, or real-time news. Giving an agent access to current web data solves this, but building an agent research loop usually introduces three problems:

  • Scattered subscriptions. You need a search provider for query results, a proxy scraper to bypass anti-bot blocks on single pages, and an extraction pipeline to clean up raw HTML into tokens an LLM can parse.
  • Key proliferation. Each external tool demands its own environment variables, dashboard, and secret rotation across development, CI, and agent environments.
  • Zero visibility into agent spend. If an autonomous loop runs into a deep recursive link cycle, an unmonitored agent can burn through balances before you notice.

With thelastapi, web research is built directly into the same platform your agent uses for databases, hosting, and AI models. One key covers everything, whether your agent runs in Claude Code, Codex, Cursor, or a background script. For a broader overview of consolidation, see one API key for an AI agent: hosting, database, email, payments and models.

Web search: finding titles, URLs, and snippets

When an agent needs to discover sources or locate current answers, it starts with web search. Rather than returning bloated HTML pages, web search returns clean structured results containing titles, URLs, and snippets.

From the terminal or a shell-enabled agent, use the vibiz search command:

Terminal
vibiz search "latest Postgres features" --limit 5

If you want machine-readable output inside an automated script, pass --json:

Terminal
vibiz search "open source LLM frameworks" --limit 5 --json

Under the hood, this executes the web.search action. You can also run the underlying action directly using vibiz run:

Terminal
vibiz run web.search --input '{"query":"developer tools roadmap 2026","limit":5}' --json

The output gives your agent precise links and summaries. The agent can then select the most relevant URLs to read in depth.

Scraping a single page into clean markdown

Raw HTML is noisy. Web pages include scripts, stylesheets, tracking tags, navigation bars, and cookie banners that waste context windows.

The vibiz scrape command takes a URL, handles page rendering, and converts the core content into structured markdown. This format fits directly into prompt contexts:

Terminal
vibiz scrape https://example.com/blog/announcement

Pass --json to receive structured metadata along with the markdown text:

Terminal
vibiz scrape https://example.com/docs/api --json

You can also execute the action directly via vibiz run web.scrape:

Terminal
vibiz run web.scrape --input '{"url":"https://example.com/pricing"}' --json

Because the output is converted to markdown before the agent reads it, token consumption stays predictable. The agent gets the headings, tables, links, and body text needed to complete its task without having to strip document boilerplate itself.

Crawling multi-page sites without managing infrastructure

When an agent needs to research an entire documentation site, knowledge base, or product catalog, one page is not enough. Manually chaining scrapers together requires writing queues, tracking visited links, and handling edge cases.

The vibiz crawl command crawls an entire site page by page, rendering each page into markdown:

Terminal
vibiz crawl https://example.com --max-pages 10

To call the crawler as an action, use web.crawl:

Terminal
vibiz run web.crawl --input '{"url":"https://example.com","max_pages":10}' --json

The action traverses internal links on the target domain up to the page limit you specify (up to 100 pages per crawl). It returns clean markdown for each discovered page, letting your agent synthesize cross-page knowledge without getting stuck in infinite loops.

ActionCommandPrimary outputBest for
web.searchvibiz search "<query>"Titles, URLs, snippetsDiscovering sources and checking current facts
web.scrapevibiz scrape <url>Markdown of a single pageReading specific articles, pricing, and documentation
web.crawlvibiz crawl <url>Markdown of up to 100 pagesResearching full sites, knowledge bases, and site architectures
web.screenshotvibiz run web.screenshotPNG image fileChecking visual page layouts and UI rendering

Connecting search and scrape tools over MCP

If your agent runs inside Claude Code, Codex, or Cursor, you do not need to wrap CLI commands in custom shell scripts. You can expose search, scrape, and crawl capabilities as native Model Context Protocol (MCP) tools. Learn more about how MCP coordinates agent tooling in what is MCP, and how do you give Claude Code real tools?.

You can connect your coding agent using the CLI:

Terminal
vibiz mcp add --clients "Claude Code"

Or connect the local MCP server with Claude Code:

Terminal
claude mcp add vibiz -- npx -y @vibiz/cli mcp

For Cursor, add the server configuration to .cursor/mcp.json:

JSON
{
  "mcpServers": {
    "vibiz": {
      "command": "npx",
      "args": ["-y", "@vibiz/cli", "mcp"],
      "env": { "VIBIZ_API_KEY": "${env:VIBIZ_API_KEY}" }
    }
  }
}

Once connected, your agent gains access to the vibiz_run_action tool. When an agent decides it needs to search the web or fetch a page, it issues a structured tool call:

JSON
{
  "action": "web.scrape",
  "input": {
    "url": "https://example.com/changelog"
  }
}

The tool executes the scrape on thelastapi's infrastructure and returns the cleaned markdown content straight to the model. You can also install the bundled agent skills so the agent knows when to invoke each workflow:

Terminal
vibiz skills add --global

Review the Agent skills documentation and the Agents guide for client-specific configuration details.

Inspecting run logs and controlling agent spend

Web research actions are metered per call. To prevent runaway spending during autonomous research tasks, thelastapi gives you complete visibility and control over execution history and budgets.

Reviewing past runs

Every web action is persisted. You can inspect previous research queries and view their raw output at any time from the terminal or in the console under Web data.

List past runs across your project:

Terminal
vibiz runs --action web.scrape --since 1d

To inspect the exact inputs and outputs returned during a specific run:

Terminal
vibiz runs run_123 --json

MCP-connected agents can also review previous results using the vibiz_action_runs tool. This prevents redundant web calls when an agent needs to re-read data it collected earlier in the same session.

Setting daily limits and permission scopes

Agents should only spend what you authorize. You can check your account's daily spend limit (which covers all projects, AI calls, and web data actions) and adjust it directly from the CLI:

Terminal
vibiz billing limit        # view the daily spending limit
vibiz billing limit 25     # set a $25 daily limit (requires full-access key)

To see what has been spent across tools and models, inspect your usage breakdown:

Terminal
vibiz usage --period 24h --by tool

When provisioning keys for automated agents, use a Limited key rather than an account-wide owner key. A Limited key runs web actions, builds projects, and manages development environments, but it can never read production secrets or move live money. For guidance on boundary isolation, read why agents should never touch production rows. Check Keys and environments and the CLI documentation for key management commands.

Terminal
vibiz keys create research-bot --limited

If a Limited key ever attempts an operation reserved for the account owner, it receives a 403 forbidden status with details.reason: "owner_only" and a direct link where you can approve the action.

Getting started

To equip your coding agent with web research tools, set up your project environment using the Quickstart guide. If you are new to the platform, create your account at thelastapi.dev/start. Then install the CLI with npm install -g @vibiz/cli, run vibiz login to authorize your computer, and run vibiz skills add --global to give your agent the tools to research, scrape, and build.

Frequently asked questions

What actions can an agent run for web research?

An agent can run `web.search` for web and news results, `web.scrape` for single-page markdown extraction, `web.crawl` for site-wide crawling up to 100 pages, and `web.screenshot` for page images.

How do web research tools appear in an MCP client?

In an MCP client like Claude Code, Codex, or Cursor, web actions are called through the `vibiz_run_action` tool, and previous run results can be inspected using `vibiz_action_runs`.

Can a Limited API key perform web scraping and searches?

Yes. Limited keys can run actions, use AI models, deploy sites, and create projects, while remaining restricted from reading production secrets, changing keys, or moving live money.

How can I prevent an autonomous agent from overspending on web searches?

You can set an account-wide daily spending limit covering all projects, AI models, and web data using `vibiz billing limit <dollars>`, and inspect spend by tool using `vibiz usage --by tool`.