I pointed raw requests at 8 anti-bot-protected sites, then tested every proxy route — here's what actually worked

Last week I needed a boring answer to a boring question: for small public-data jobs, do you actually need a residential proxy pool, or is it cargo cult? So I probed 8 well-known bot-protected sites directly, then ran every routing option I have access to on the Apify platform, and wrote down the HTTP codes. No vendor marketing — just results. Every number in this post comes from two dated test runs we keep in our repo.

Spoiler ordering, because every tutorial should include it: the official API option came first for a reason, and it should stay first.

The 8-site direct test (2026-11-05)

From a plain home lab connection — no proxy, a normal browser User-Agent, one GET per site:

Two important caveats before you quote a 200 in a board meeting:

• A 200 on a landing page is not a 200 on the data you want. Instagram and X return HTML shells; the interesting content sits behind logged-in sessions. That's an auth problem, not a proxy problem, and no pool solves it cleanly.
• "Cloudflare" is not one thing. hypixel.net runs Cloudflare and served the real page. Glassdoor runs Cloudflare and challenged me at the edge. The vendor matters less than each site's own ruleset.

I also re-ran the same 8 sites with the User-Agent set to python-urllib/3.14 — the default-your-library-sends giveaway. Only one result changed: zillow flipped from 200 to 403. UA blocking is mostly dead as a first-line defense; it's IP reputation now.

Then the proxy test: Reddit vs. every route (2026-11-03, cloud infrastructure)

The reason Reddit is the interesting row: we test our two Reddit actors' proxy router against real buckets. On Apify's infrastructure we tried the unauthenticated public-JSON endpoint (reddit.com/r/all/top.json-style) through every routing option:

Five configurations, one outcome. Reddit blocks reputable shared pools wholesale, including residential-class shared pools — the pool's reputation is the problem, not the protocol or the UA.

Where residential proxies actually help

So the honest ladder for a small public-data job looks like this:

1. Official API first. Reddit's Data API with your own free app credentials works from any IP and is the supported path. Most of the "proxies for everything" reflex dies right here — a credential beats rotating through 500 IPs that are all on a blocklist.
2. Your own premium residential pool, for the sites without an API that also edge-block datacenter IPs. Same config, same rotation logic — it just exits from cleaner ranges. In our router, a buyer-supplied pool slots into a proxyList input and gets round-robin, per-proxy cooldowns (30s base, doubling to a 10-minute cap), and a blacklist after 5 consecutive failures or a 407. Those failures get loud instead of silently becoming empty datasets.
3. Shared pools are a coin flip on the hard sites. Shared datacenter and shared residential ranges carry other people's traffic history. Reddit says 403 to all of Apify's; other sites vary.

And one behavior we added out of frustration: if an actor notices that every request in a run was refused, it tells you exactly which routes were tried and charges nothing. A silent zero-item dataset is the worst possible proxy-test result.

The probe script

The whole direct test is un-pretty urllib — which is the point. If your pipeline fails with this, a smarter fetcher won't save you; you need a different route, not a different client:

import urllib.request, urllib.error SITES = [ "https://hypixel.net/forums/", "https://www.instagram.com/", "https://x.com/", "https://www.nike.com/", "https://www.zillow.com/", "https://www.reddit.com/r/all/top.json?t=day&limit=10", "https://www.glassdoor.com/", "https://www.yelp.com/", ] UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/130.0.0.0 Safari/537.36") for url in SITES: req = urllib.request.Request(url, headers={"User-Agent": UA}) try: r = urllib.request.urlopen(req, timeout=25) print(f"{url:50s} {r.status}") except urllib.error.HTTPError as e: print(f"{url:50s} {e.code}") 

Run it twice — once with your real UA, once with python-urllib/3.14 — and you've reproduced all four findings of this post in about four seconds.

Disclosures

I run OFFGRID LABS, a small shop that sells public-data actors on the Apify Store, and the proxy/router numbers above come from the PROXY.md test logs of our two Reddit actors. The Reddit app-credential path and the bring-your-own-proxyList escape hatch live in offgrid-labs/reddit-scraper — linked once because it's the thing that generated the data. Every claim above is in the actor's documentation, complete with UTC dates, because our content rules say a number without a source file is a rumor.

Takeaway: check whether the site has an official API before you buy proxies. If it does, that's the whole route. If it doesn't, expect datacenter IPs to be dead on popular targets, expect shared residential to inherit strangers' reputations, and expect a good pool to still fail sometimes — so measure per-proxy health and fail loudly, not empty.