How To Make Your First Web Data API Request

AI agents and RAG pipelines need web pages they can read. A plain HTTP request to a protected site can get a "Robot or human?" page instead of the content. Other pages arrive without their text, because it loads later with JavaScript. The Decodo Web Data API fetches those pages and returns them as Markdown or HTML.

This guide goes from sign-up to a working Python script in about 5 minutes. You'll send your first request, learn to read the response, and finish with a script that checks each page for the content your agent expects.

TL;DR

• Activate the Decodo Web Data API Free plan, copy your API key from the Decodo dashboard, and export it as the DECODO_API_KEY environment variable.
• Send a POST request to https://data.decodo.com/v1/scrape with your key in an Authorization: Bearer header and {"url": "..."} as the JSON body. The response returns the page in results[0].content and the target's own status code in results[0].status_code, so your code can check whether the page loaded.
• The same key and the same Authorization header work on /search, /map, and /crawl.

What you'll need

• Python 3, tested with 3.10 and 3.15.
• The requests library, tested with 2.34.
• A Decodo account with the Web Data API Free plan activated.
• A terminal.

The Web Data API in 60 seconds

The Web Data API gives you /scrape, /search, /map, and /crawl with a single API key. Pick one based on what your agent needs:

Proxy rotation, anti-bot handling, and JavaScript rendering run on our servers, so your code stays a few lines long. Every endpoint runs on Decodo's proxy network of 125M+ IPs, with a 99.99% success rate.

The difference is clearest on protected sites. On a Walmart search URL, /scrape returned the real results page in all 10 attempts. Plain requests calls to the same URL from a home connection got HTTP 200 and a "Robot or human?" page every time. With headers copied from Chrome, requests got the real page in 5 of the 10 attempts.

Step 1: Get your API key

Sign up, then open Web Data API in the Decodo dashboard. On the Playground tab, click Activate free plan:

The pricing page lists what the Free plan includes. Once the plan card shows ACTIVE, copy the API key from the top right of the same tab:

Run the command for your shell, then paste the key. That way, the key isn't saved in your code or your shell history. On macOS and Linux, nothing appears while you paste. On Windows, the key is visible as you type it. Press Enter, then run the scripts from the same terminal:

# macOS and Linux read -rs DECODO_API_KEY && export DECODO_API_KEY # Windows PowerShell $env:DECODO_API_KEY = Read-Host "Paste your API key" 

Step 2: Send your first request

The target, https://ip.decodo.com/json, returns the IP and country that a request came from. Save this script as first_request.py:

import json import os import sys import requests api_key = os.environ.get("DECODO_API_KEY") if not api_key: sys.exit("DECODO_API_KEY is not set. Export your Web Data API key first.") response = requests.post( "https://data.decodo.com/v1/scrape", headers={"Authorization": f"Bearer {api_key}"}, json={"url": "https://ip.decodo.com/json"}, timeout=180, # in an agent tool, keep this below the tool's time limit ) response.raise_for_status() result = response.json()["results"][0] ip_info = json.loads(result["content"]) print("target status code:", result["status_code"]) print("exit IP:", ip_info["proxy"]["ip"], ip_info["country"]["code"]) 

When you run python first_request.py, it prints the IP address and country of the proxy that sent the request:

target status code: 200 exit IP: 38.127.100.236 US 

From a macOS or Linux terminal with cURL 7.82 or later, run the same request:

curl https://data.decodo.com/v1/scrape \ -H "Authorization: Bearer $DECODO_API_KEY" \ --json '{"url": "https://ip.decodo.com/json"}' 

That's the whole request.

Step 3: Read the response

Here's what step 2's request to https://ip.decodo.com/json returned, shortened:

{ "results": [ { "content": "{\n \"browser\": {\n \"name\": \"Firefox\", ...", "type": "raw", "headers": {"content-type": "application/json", "server": "nginx"}, "cookies": [], "status_code": 200, "task_id": "7518637022634883073" } ] } 

Read the target's status code inside the API's HTTP 200 response. The API returns HTTP 200 whenever it has processed your request, so an HTTP 200 response can mean 3 different things:

• The page loaded. results[0].status_code is 200, and the content contains the page.
• The target returned an error page. For a missing GitHub repo, results[0].status_code was 404, with GitHub's 404 page as content.
• The target returned an error with no page. https://httpbin.org/status/403 is a test URL that always returns 403 with an empty page. The API still returned HTTP 200. The response body had "status": "failed", "status_code": 613, and no results array. These top-level fields describe the API call itself, while results[0].status_code describes the target's response.

So your code should check the HTTP status code, the status field, and results[0].status_code.

You pay only for requests that return a result, as the billing page explains. A target's error page with content, like the GitHub 404, is a result.

Step 4: Change 1 parameter to render JavaScript

The practice site https://quotes.toscrape.com/js/ loads its quotes with JavaScript. Save this as render.py. The script sends 2 requests for Markdown, and only the second enables rendering with "headless": "html":

import os import requests API_URL = "https://data.decodo.com/v1/scrape" HEADERS = {"Authorization": f"Bearer {os.environ['DECODO_API_KEY']}"} for label, extra in (("no rendering", {}), ("rendered", {"headless": "html"})): payload = {"url": "https://quotes.toscrape.com/js/", "markdown": True, **extra} response = requests.post(API_URL, headers=HEADERS, json=payload, timeout=180) response.raise_for_status() result = response.json()["results"][0] quotes = result["content"].count("“") # each quote starts with a curly “  status, kind = result["status_code"], result["type"] print(f"{label}: target status code {status}, type {kind}, {quotes} quotes") 

When you run python render.py, it prints:

no rendering: target status code 200, type markdown, 0 quotes rendered: target status code 200, type markdown, 10 quotes 

Without rendering, the HTTP status code and the target status code both show success. Only the quote count shows the content is missing. The request with "headless": "html" returns all 10. With "markdown": true, the rendered page is still returned as Markdown, as the type markdown output confirms.

Enable rendering for pages that load their content with JavaScript. Rendering runs a headless browser, so it costs more than a request without it.

Markdown, proxy pools, and other options

Markdown keeps the structure a model needs in far fewer tokens than HTML. With OpenAI's o200k_base tokenizer, the Wikipedia article on web scraping was about 72,000 tokens as HTML and about 16,500 as Markdown. Plain text from Beautiful Soup's get_text() was smaller, but it lost the headings and links that Markdown keeps.

Try the standard proxy pool on sites without bot protection. Send "proxy_pool": "standard", which costs less. By default, requests use the premium pool, which is built for protected sites. If the content you expect is missing on the standard pool, switch to "proxy_pool": "premium".

Other options, such as geo for the proxy location, are in the parameter reference.

Handle errors and retries

Some errors require a change to your request, while others only require a retry. The table lists the common ones from the status codes page:

A 401 can also mean the request went to another API's host. Examples written for Decodo's Web Scraping API use scraper-api.decodo.com and that API's token. For the Web Data API, use your Web Data API key with data.decodo.com. The migration guide explains how to move an existing Web Scraping API or Fast Search API setup to the Web Data API when you're ready.

For production, save the script below as scrape.py. It checks each response in this order:

The Retry class from urllib3 retries 429 and common 5xx errors when allowed_methods includes POST. A 612 or 613 means our servers have already retried, so scrape() changes the request instead of resending it. A 613 isn't charged, so you pay only when an attempt returns a result:

import os import sys import requests from requests.adapters import HTTPAdapter from urllib3.util import Retry API_URL = "https://data.decodo.com/v1/scrape" session = requests.Session() session.headers["Authorization"] = f"Bearer {os.environ['DECODO_API_KEY']}" retry = Retry( total=2, read=False, # don't resend a request that timed out or lost its connection  backoff_factor=2, status_forcelist=[429, 500, 502, 503, 504, 524], allowed_methods={"POST"}, raise_on_status=False, ) session.mount("https://", HTTPAdapter(max_retries=retry)) def scrape(payload, expect=""): for attempt in (1, 2): response = session.post(API_URL, json=payload, timeout=180) try: body = response.json() except ValueError: body = {} if response.ok and body.get("status") != "failed" and body.get("results"): result = body["results"][0] if expect in result["content"] or result["status_code"] >= 400: return result status = result["status_code"] note = f"target status code {status}, {expect!r} not found" print(f"attempt {attempt}: {note}") else: code = body.get("status_code", response.status_code) message = body.get("message", response.text[:200]) task_id = body.get("task_id", "-") print(f"attempt {attempt}: {code} {message} (task_id {task_id})") if code not in (612, 613): return None payload = {**payload, "headless": "html"} return None if len(sys.argv) < 2: sys.exit("Usage: python scrape.py URL [EXPECTED_TEXT]") url = sys.argv[1] expect = sys.argv[2] if len(sys.argv) > 2 else "" result = scrape({"url": url, "markdown": True}, expect) if result is None: sys.exit(1) if result["status_code"] >= 400: print("the target returned", result["status_code"]) elif expect: print(result["content"].count(expect), "matches for", repr(expect)) print(len(result["content"]), "characters of Markdown") 

Run it on the quotes page from Step 4, and pass the curly quote mark (“) as the text the page should contain. Every quote on that page starts with this mark. After the first attempt, the script enables rendering:

$ python scrape.py https://quotes.toscrape.com/js/ "“" attempt 1: target status code 200, '“' not found 10 matches for '“' 1567 characters of Markdown 

Use the same key on /search, /map, and /crawl

Agents often search first, then scrape only the result they pick. The next script follows that pattern, then runs /map and /crawl on a docs site. Every request uses the same header. Save it as endpoints.py:

import os import time import requests BASE_URL = "https://data.decodo.com/v1" HEADERS = {"Authorization": f"Bearer {os.environ['DECODO_API_KEY']}"} def post(endpoint, payload): url = f"{BASE_URL}/{endpoint}" response = requests.post(url, headers=HEADERS, json=payload, timeout=180) response.raise_for_status() return response.json() def get(endpoint): url = f"{BASE_URL}/{endpoint}" response = requests.get(url, headers=HEADERS, timeout=30) response.raise_for_status() return response.json() search = post("search", {"query": "mcp server oauth"}) top_link = search["organic"][0]["link"] page = post("scrape", {"url": top_link, "markdown": True})["results"][0] site = top_link.split("/")[2] print("search: top result on", site, "| target status code", page["status_code"]) site_map = post("map", {"url": "https://help.decodo.com"}) print("map:", len(site_map["results"][0]["content"]["links"]), "URLs") crawl_id = post("crawl", {"url": "https://help.decodo.com", "limit": 5})["crawl_id"] job = get(f"crawl/{crawl_id}/status") while job["status"] == "pending": # every other status means the crawl has ended  time.sleep(10) job = get(f"crawl/{crawl_id}/status") pages = get(f"crawl/{crawl_id}?type=content")["url"] # "url" holds the page list print("crawl:", job["status"], "|", len(pages), "pages, first:", pages[0]["url"]) 

The crawl runs as a background job. The script checks its status every 10 seconds and fetches the crawled pages once the crawl ends. When you run python endpoints.py, it prints output like this. Your results and counts will differ as the sites change:

search: top result on modelcontextprotocol.io | target status code 200 map: 801 URLs crawl: done | 5 pages, first: https://help.decodo.com/ 

For larger crawls, follow the next link in the response to get the remaining pages, as the crawl results docs explain. For the full list of fields that each endpoint returns, see the /search docs, /map docs, and /crawl docs.

Next steps: MCP, LangChain, LlamaIndex, and n8n

Every later request works like the first one in step 2. It uses the same key, header, and JSON body across all 4 endpoints.

To give your agent our web data tools over the Model Context Protocol (MCP), connect it to our hosted Source：https://dev.to/decodo_official/how-to-make-your-first-web-data-api-request-ool