# Give an agent web research tools: search, scrape and crawl

> Give your AI agent web search, page scraping, and site crawling with one key using thelastapi via the CLI, Model Context Protocol, or HTTP.

**You can give an AI agent web search, page scraping, and multi-page crawling through one key using thelastapi's built-in actions and Model Context Protocol (MCP) server.** Instead of setting up separate search APIs, headless browser scrapers, and crawling fleets, your agent can search query results, turn web pages into clean markdown, and crawl whole domains using unified CLI commands or MCP tools. Every run is recorded in your project history, billed against your account credits, and capped by your daily spend limit.

This guide shows how web research tools work for agents, how to invoke them from terminal workflows or MCP clients, and how to monitor past runs.

## Why web research needs dedicated tools for agents

Language models make mistakes when guessing live URLs, outdated facts, or real-time news. Giving an agent access to current web data solves this, but building an agent research loop usually introduces three problems:

- **Scattered subscriptions.** You need a search provider for query results, a proxy scraper to bypass anti-bot blocks on single pages, and an extraction pipeline to clean up raw HTML into tokens an LLM can parse.
- **Key proliferation.** Each external tool demands its own environment variables, dashboard, and secret rotation across development, CI, and agent environments.
- **Zero visibility into agent spend.** If an autonomous loop runs into a deep recursive link cycle, an unmonitored agent can burn through balances before you notice.

With thelastapi, web research is built directly into the same platform your agent uses for databases, hosting, and AI models. One key covers everything, whether your agent runs in Claude Code, Codex, Cursor, or a background script. For a broader overview of consolidation, see [one API key for an AI agent: hosting, database, email, payments and models](https://thelastapi.dev/blog/one-api-key-for-ai-agent).

## Web search: finding titles, URLs, and snippets

When an agent needs to discover sources or locate current answers, it starts with web search. Rather than returning bloated HTML pages, web search returns clean structured results containing titles, URLs, and snippets.

From the terminal or a shell-enabled agent, use the `vibiz search` command:

```bash
vibiz search "latest Postgres features" --limit 5
```

If you want machine-readable output inside an automated script, pass `--json`:

```bash
vibiz search "open source LLM frameworks" --limit 5 --json
```

Under the hood, this executes the `web.search` action. You can also run the underlying action directly using `vibiz run`:

```bash
vibiz run web.search --input '{"query":"developer tools roadmap 2026","limit":5}' --json
```

The output gives your agent precise links and summaries. The agent can then select the most relevant URLs to read in depth.

## Scraping a single page into clean markdown

Raw HTML is noisy. Web pages include scripts, stylesheets, tracking tags, navigation bars, and cookie banners that waste context windows. 

The `vibiz scrape` command takes a URL, handles page rendering, and converts the core content into structured markdown. This format fits directly into prompt contexts:

```bash
vibiz scrape https://example.com/blog/announcement
```

Pass `--json` to receive structured metadata along with the markdown text:

```bash
vibiz scrape https://example.com/docs/api --json
```

You can also execute the action directly via `vibiz run web.scrape`:

```bash
vibiz run web.scrape --input '{"url":"https://example.com/pricing"}' --json
```

Because the output is converted to markdown before the agent reads it, token consumption stays predictable. The agent gets the headings, tables, links, and body text needed to complete its task without having to strip document boilerplate itself.

## Crawling multi-page sites without managing infrastructure

When an agent needs to research an entire documentation site, knowledge base, or product catalog, one page is not enough. Manually chaining scrapers together requires writing queues, tracking visited links, and handling edge cases.

The `vibiz crawl` command crawls an entire site page by page, rendering each page into markdown:

```bash
vibiz crawl https://example.com --max-pages 10
```

To call the crawler as an action, use `web.crawl`:

```bash
vibiz run web.crawl --input '{"url":"https://example.com","max_pages":10}' --json
```

The action traverses internal links on the target domain up to the page limit you specify (up to 100 pages per crawl). It returns clean markdown for each discovered page, letting your agent synthesize cross-page knowledge without getting stuck in infinite loops.

| Action | Command | Primary output | Best for |
| --- | --- | --- | --- |
| `web.search` | `vibiz search "<query>"` | Titles, URLs, snippets | Discovering sources and checking current facts |
| `web.scrape` | `vibiz scrape <url>` | Markdown of a single page | Reading specific articles, pricing, and documentation |
| `web.crawl` | `vibiz crawl <url>` | Markdown of up to 100 pages | Researching full sites, knowledge bases, and site architectures |
| `web.screenshot` | `vibiz run web.screenshot` | PNG image file | Checking visual page layouts and UI rendering |

## Connecting search and scrape tools over MCP

If your agent runs inside Claude Code, Codex, or Cursor, you do not need to wrap CLI commands in custom shell scripts. You can expose search, scrape, and crawl capabilities as native Model Context Protocol (MCP) tools. Learn more about how MCP coordinates agent tooling in [what is MCP, and how do you give Claude Code real tools?](https://thelastapi.dev/blog/what-is-mcp-claude-code-tools).

You can connect your coding agent using the CLI:

```bash
vibiz mcp add --clients "Claude Code"
```

Or connect the local MCP server with Claude Code:

```bash
claude mcp add vibiz -- npx -y @vibiz/cli mcp
```

For Cursor, add the server configuration to `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "vibiz": {
      "command": "npx",
      "args": ["-y", "@vibiz/cli", "mcp"],
      "env": { "VIBIZ_API_KEY": "${env:VIBIZ_API_KEY}" }
    }
  }
}
```

Once connected, your agent gains access to the `vibiz_run_action` tool. When an agent decides it needs to search the web or fetch a page, it issues a structured tool call:

```json
{
  "action": "web.scrape",
  "input": {
    "url": "https://example.com/changelog"
  }
}
```

The tool executes the scrape on thelastapi's infrastructure and returns the cleaned markdown content straight to the model. You can also install the bundled agent skills so the agent knows when to invoke each workflow:

```bash
vibiz skills add --global
```

Review the [Agent skills documentation](https://thelastapi.dev/docs/skills) and the [Agents guide](https://thelastapi.dev/docs/agents) for client-specific configuration details.

## Inspecting run logs and controlling agent spend

Web research actions are metered per call. To prevent runaway spending during autonomous research tasks, thelastapi gives you complete visibility and control over execution history and budgets.

### Reviewing past runs

Every web action is persisted. You can inspect previous research queries and view their raw output at any time from the terminal or in the console under **Web data**.

List past runs across your project:

```bash
vibiz runs --action web.scrape --since 1d
```

To inspect the exact inputs and outputs returned during a specific run:

```bash
vibiz runs run_123 --json
```

MCP-connected agents can also review previous results using the `vibiz_action_runs` tool. This prevents redundant web calls when an agent needs to re-read data it collected earlier in the same session.

### Setting daily limits and permission scopes

Agents should only spend what you authorize. You can check your account's daily spend limit (which covers all projects, AI calls, and web data actions) and adjust it directly from the CLI:

```bash
vibiz billing limit        # view the daily spending limit
vibiz billing limit 25     # set a $25 daily limit (requires full-access key)
```

To see what has been spent across tools and models, inspect your usage breakdown:

```bash
vibiz usage --period 24h --by tool
```

When provisioning keys for automated agents, use a Limited key rather than an account-wide owner key. A Limited key runs web actions, builds projects, and manages development environments, but it can never read production secrets or move live money. For guidance on boundary isolation, read [why agents should never touch production rows](https://thelastapi.dev/blog/agents-never-touch-production-rows). Check [Keys and environments](https://thelastapi.dev/docs/keys) and the [CLI documentation](https://thelastapi.dev/docs/cli) for key management commands.

```bash
vibiz keys create research-bot --limited
```

If a Limited key ever attempts an operation reserved for the account owner, it receives a `403 forbidden` status with `details.reason: "owner_only"` and a direct link where you can approve the action.

## Getting started

To equip your coding agent with web research tools, set up your project environment using the [Quickstart guide](https://thelastapi.dev/docs/quickstart). If you are new to the platform, create your account at [thelastapi.dev/start](https://thelastapi.dev/start). Then install the CLI with `npm install -g @vibiz/cli`, run `vibiz login` to authorize your computer, and run `vibiz skills add --global` to give your agent the tools to research, scrape, and build.

## Frequently asked questions

### What actions can an agent run for web research?

An agent can run `web.search` for web and news results, `web.scrape` for single-page markdown extraction, `web.crawl` for site-wide crawling up to 100 pages, and `web.screenshot` for page images.

### How do web research tools appear in an MCP client?

In an MCP client like Claude Code, Codex, or Cursor, web actions are called through the `vibiz_run_action` tool, and previous run results can be inspected using `vibiz_action_runs`.

### Can a Limited API key perform web scraping and searches?

Yes. Limited keys can run actions, use AI models, deploy sites, and create projects, while remaining restricted from reading production secrets, changing keys, or moving live money.

### How can I prevent an autonomous agent from overspending on web searches?

You can set an account-wide daily spending limit covering all projects, AI models, and web data using `vibiz billing limit <dollars>`, and inspect spend by tool using `vibiz usage --by tool`.

---

Published 2026-11-15 · https://thelastapi.dev/blog/agent-web-research-tools
