Free MCP Web Scraping for AI Agents: Search, Extract and Render With No API Key

Most "web scraping MCP" setups start the same way: create an account, copy an API key into a config file, pick a plan. That's a lot of friction just to find out whether your agent can read a page.

The AgentSearch MCP server skips all of it. One npx command gives Claude, Cursor, Claude Code, VS Code, Cline or any other MCP client three web tools that work with nothing configured:

Web search: free, up to 25 searches a day per IP
Extract (any URL to clean markdown): free trial, 5 calls a day per IP
Render (JavaScript-heavy pages in headless Chromium): free trial, 3 calls a day per IP

No API key, no signup, no wallet. When you outgrow the free allowance, the same server pays per call over x402 from a small USDC wallet on Base, and you never create an account.

This guide covers the install for each client, how the three tools fit together, a script that tests the server without any client, and what happens when you hit the daily limits.

Install: one command, any MCP client

The server is on npm as @agentsearchhq/agentsearch-mcp. It runs over stdio, so your client starts it as a local process. You need Node.js 20 or newer.

npx -y @agentsearchhq/agentsearch-mcp

Claude Code

claude mcp add agentsearch -- npx -y @agentsearchhq/agentsearch-mcp

Add --scope project to write it to .mcp.json so everyone who clones the repo gets it. More detail in Add web search to Claude Code.

Cursor

Put this in ~/.cursor/mcp.json (global) or .cursor/mcp.json (one project):

{ "mcpServers": { "agentsearch": { "command": "npx", "args": ["-y", "@agentsearchhq/agentsearch-mcp"] } } }

See Add web search to Cursor for the full walkthrough.

Claude Desktop

Same block, in claude_desktop_config.json (Settings, then Developer, then Edit Config). Restart Claude Desktop afterwards. Guide: Claude Desktop MCP web search.

VS Code (Copilot agent mode)

VS Code uses a servers key in .vscode/mcp.json:

{ "servers": { "agentsearch": { "type": "stdio", "command": "npx", "args": ["-y", "@agentsearchhq/agentsearch-mcp"] } } }

Cline, Windsurf and other clients take the same command and args. On Windows, if npx isn't found, use "command": "cmd" with "args": ["/c", "npx", "-y", "@agentsearchhq/agentsearch-mcp"].

When the server starts with nothing set, it prints a line on stderr confirming the mode: search on the free tier, extract and render on the free trial.

The tools your agent gets

After the install, your client lists these tools:

Tool	What it does	Free with no config?
`agentsearch_web_search`	Live web search, up to 5 results per call	Yes, up to 25/day per IP
`agentsearch_extract`	Fetch a URL and return clean markdown, title, links and meta	Yes, 5/day per IP (trial)
`agentsearch_render`	Render a JavaScript-heavy page in headless Chromium, then return markdown or text	Yes, 3/day per IP (trial, text output, no screenshots)
`agentsearch_search_and_read`	Search plus the top pages as markdown in one paid call	No, needs a wallet
`agentsearch_review_lookup` / `agentsearch_review_submit`	Check which web services work right now, and leave a review	Yes

The pattern that works best for research is a ladder, not a single tool:

Search to find candidate URLs (free).
Extract the one or two best URLs to read the actual page, not just the snippet.
Render only when Extract comes back empty or partial, which usually means the page is a JavaScript app.

Snippets tell your agent a page exists. Extract tells it what the page says. That difference is why "read the page" is the step that changes answer quality, and why the trial covers it.

Tell your agent how to use the ladder

Agents do better with a short instruction than with a pile of tools. Drop something like this into CLAUDE.md, Cursor rules, or your system prompt:

For questions that need current information: 1. Call agentsearch_web_search first. 2. Pick the 1-2 most relevant URLs and call agentsearch_extract on them. 3. If extract returns empty or very short markdown, call agentsearch_render on that URL. 4. Cite the URLs you actually read. Don't extract every result.

The last line matters on the free trial. Five extracts a day goes a long way if your agent reads the best two pages per question, and runs out fast if it reads all five search results every time.

Test the server without any client

Before wiring it into an editor, you can check that the tools respond. This small Node script starts the server, lists its tools and runs one free extract over the MCP protocol:

// test-agentsearch-mcp.mjs (run: node test-agentsearch-mcp.mjs) import { spawn } from "node:child_process"; const server = spawn("npx", ["-y", "@agentsearchhq/agentsearch-mcp"], { stdio: ["pipe", "pipe", "inherit"], }); let buffer = ""; const pending = {}; server.stdout.on("data", (chunk) => { buffer += chunk; let i; while ((i = buffer.indexOf("\n")) >= 0) { const line = buffer.slice(0, i); buffer = buffer.slice(i + 1); try { const msg = JSON.parse(line); if (pending[msg.id]) { pending[msg.id](msg); delete pending[msg.id]; } } catch {} } }); let nextId = 0; const rpc = (method, params) => new Promise((resolve) => { const id = ++nextId; pending[id] = resolve; server.stdin.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n"); }); await rpc("initialize", { protocolVersion: "2025-06-18", capabilities: {}, clientInfo: { name: "smoke-test", version: "0.0.1" }, }); server.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n"); const tools = await rpc("tools/list", {}); console.log("tools:", tools.result.tools.map((t) => t.name).join(", ")); const res = await rpc("tools/call", { name: "agentsearch_extract", arguments: { url: "https://example.com", max_chars: 500 }, }); const page = JSON.parse(res.result.content[0].text); console.log(page.title, "|", page.markdown.slice(0, 120)); console.log("trial:", page.free_trial); server.kill();

A successful extract returns the page's title, markdown, links and a meta block with the HTTP status and character count. On the trial it also includes a free_trial object, so you can see usage without guessing:

"free_trial": { "tool": "extract", "used_today": 1, "limit_per_ip": 5, "remaining_today": 4, "resets_at": "2026-11-12T00:00:00Z", "charged": false }

The counter resets daily at midnight UTC. Your agent can read remaining_today and decide whether a page is worth one of its reads.

When Extract comes back empty: Render

Extract fetches the HTML a server sends. That's fast and works for most docs, blogs, news and product pages. Single-page apps are different: the first HTML response is often an empty shell, and the content appears only after JavaScript runs.

agentsearch_render loads the page in headless Chromium, waits for it to settle and returns the rendered content as markdown or text. Useful options:

wait_until: domcontentloaded, load or networkidle
wait_for_selector: wait until a specific element exists
block: skip images, media, fonts or stylesheets to keep it quick
max_chars: cap how much text comes back

Target-site problems (DNS errors, timeouts, bot walls) come back as a normal result with an error object holding code, message and retryable, so your agent can move to the next URL instead of crashing. More on that pattern in Read JavaScript pages from an AI agent and on the Render page.

What happens at the free limit

The free allowance is real but small on purpose: 25 searches, 5 extracts and 3 renders a day per IP, plus shared daily capacity on top of that. When a limit is reached, the tool returns HTTP 429 with a message that explains how to keep going, so your agent (and you) can see why a call stopped.

Then you have two choices. You can wait for the daily reset. Or you can switch on pay per call.

Going past the trial: pay per call over x402

Add a wallet key to the server's env and the same tools start paying AgentSearch's direct x402 API at api.agentsearchhq.com on Base, in USDC, per call. There's still no account, no API key and no monthly plan, and you don't need ETH for gas.

{ "mcpServers": { "agentsearch": { "command": "npx", "args": ["-y", "@agentsearchhq/agentsearch-mcp"], "env": { "AGENTSEARCH_WALLET_PRIVATE_KEY": "0xYOUR_DEDICATED_WALLET_KEY", "AGENTSEARCH_MAX_TOTAL_ATOMIC": "500000" } } } }

With a wallet key set:

Extract costs $0.007 per call (about 140 pages for $1).
Render is paid per call too; see Render and pricing for the current rate.
Search stays on the free tier unless you also set AGENTSEARCH_MODE=paid, which makes it $0.008 per call.
agentsearch_search_and_read becomes available: search plus the top pages as markdown for $0.015 in one payment. Details on Search + Read.

Calls are charged only when a usable answer comes back, so a failed or empty fetch costs nothing.

Keep spend on a leash

Two caps protect the wallet:

AGENTSEARCH_MAX_TOTAL_ATOMIC limits total spend per server session in USDC atomic units (1,000,000 = $1.00, which is the default). The example above caps it at $0.50.
AGENTSEARCH_MAX_PER_CALL_ATOMIC lowers the most any single call can sign.

Use a new, dedicated wallet that holds only a few dollars of USDC on Base, never your main wallet. Don't commit the key; in project configs, reference an environment variable instead. The wallet setup guide walks through creating and funding one.

If you'd rather write the payment code yourself, the x402 Python tutorial and x402 pay-per-call guide call the same API directly.

Troubleshooting

The tools don't show up. Restart the client after editing the config. In Claude Code, run claude mcp list to check the server is connected.

npx hangs on first run. The first start downloads the package. Run npx -y @agentsearchhq/agentsearch-mcp once in a terminal, then restart your client.

Extract returns very little text. The page is probably rendered by JavaScript. Try agentsearch_render on the same URL.

You see a 429. You've hit a daily free limit or shared capacity. Wait for the reset at midnight UTC, or add a wallet key for pay per call.

You want the trial off. Set AGENTSEARCH_FREE_TRIAL=0. Extract and Render will then return setup instructions instead of making a network call until you add a wallet.

FAQ

Is this really free with no API key?

Yes. With nothing configured, search is free up to 25 calls a day per IP, and Extract and Render run on a free trial of 5 and 3 calls a day per IP. There's no signup and no key.

Which MCP clients work?

Any client that runs stdio MCP servers: Claude Code, Claude Desktop, Cursor, VS Code with Copilot, Cline, Windsurf and others.

How many results does search return?

Up to 5 per call (max_results, default 5).

Do I have to use MCP?

No. There are framework packages too, like langchain-agentsearch for LangChain in Python and an n8n community node. See the LangChain search-then-read guide and the n8n read-pages guide.

Try it now

npx -y @agentsearchhq/agentsearch-mcp

Add it to your client, ask your agent a question that needs today's web, and watch it search, read the best page and cite it. Search is free, Extract and Render are free for 5 and 3 calls a day, and when you need more, it's pay per call over x402. More setup options are on the MCP page.