Request the same product page from a US server and from a German residential IP, and you will often get two different pages. The price may be in euros, an item may be out of stock, the search results may be reordered, or you may be redirected to /de/. If your scraper runs from a US cloud box, you only ever see the US version, which is no help when you need what a shopper in Berlin or Mumbai sees.
This post covers how sites guess your location, how country- and city-level proxy targeting works, and the Python code (requests, httpx, and Playwright) for getting geo-specific content reliably. The ideas apply to any language.
What changes by location
- Prices and currency change most frequently, and some retailers run different promotions in different regions.
- Stock and delivery windows will vary, and some pages will state that a product is not available in your country.
- Search results vary by country and indeed by city in many cases (particularly for local queries such as 'pharmacy open now').
- You may be redirected to a language path such as
/de/or/fr/, or see translated listings. - Ads and sponsored placements are often targeted by region.
- Legal gating differs too, for example, EU cookie banners, age gates and licensed-content blocks.
If you want data that represents what users in X see, you have to request it from X.
How websites decide where you are
The main signal is your IP address. The server looks it up in a geo-IP database and maps it to a country, region, and city. Your proxy controls this signal and nothing else. The rest is supporting evidence:
-
Accept-Languagehints at language rather than location, but a mismatch looks odd. - A saved ship-to country, currency selector, or ZIP code from an earlier visit can override the IP entirely.
- CDNs such as Cloudflare can pass a country header like CF-IPCountry to the origin. It is derived from your IP, so you can’t set it yourself.
- In a real browser, JavaScript can read the timezone and locale, and the Geolocation API if the user allows it.
- The URL itself can carry the location, like
?gl=de&hl=deon Google or an/en-gb/prefix.
When these disagree, say a German IP with an en-US locale and a Los Angeles timezone, some sites serve the wrong version and anti-bot systems get suspicious.
Proxy types and geo accuracy
| Type | Where the IP comes from | Geo-targeting notes |
|---|---|---|
| Datacenter | Cloud and hosting providers | Cheap and fast. Country targeting is usually good; city targeting is limited to where the provider has facilities. Easy for sites to identify as non-residential. |
| Residential | Consumer ISP connections | Widest country and city coverage, and looks like a real user. Slower, and usually priced by bandwidth. |
| Mobile | Carrier networks | Useful for mobile-only content. Carrier IPs are often shared and geolocated loosely. |
| ISP (static residential) | ISP-assigned IPs hosted in data centers | Stable IPs with residential reputation, but a smaller location pool. |
For country-level scraping of tolerant sites, datacenter proxies are usually enough. Move to residential for city-level work, local SERPs, or sites with aggressive bot protection.
Country-level vs city-level targeting
Country-level targeting is the straightforward case. You ask the provider for an exit IP in a particular country, and the geo-IP databases all agree on the country virtually every time.
City-level is where it starts to break down. The provider may say an IP is in Chicago; the target site may be using a different geo-IP vendor which puts it in a different city (or indeed just the right state). Each vendor has its own estimate, and they disagree much more at the city level than at the country level. In practice:
- Treat a city target as “probably this city” and then check what the target actually shows.
- Have a fallback ready: try the city, then the state/region, then the country.
- For zip-specific content (e.g., grocery delivery, local inventory), an IP alone is often not sufficient. A lot of sites will ask for a zip and then store that in a cookie or on the account, so you'll need to set that explicitly.
How providers expose targeting
Every provider does it its own way, so be sure to read their documentation. The common patterns are:
- Parameters in the proxy username, for example,
user-country-de-city-berlin. - Per-country hostnames or ports, such as
de.gateway.example.com:8000. - A header or API parameter on the provider’s scraping API.
- A fixed IP list that you filter by location metadata yourself.
The code below puts all of that in one function, so switching providers means changing one place. The format string is a placeholder.
from dataclasses import dataclass from typing import Optional @dataclass(frozen=True) class GeoTarget: country: str # ISO 3166-1 alpha-2, e.g. "DE" city: Optional[str] = None # e.g. "berlin" session: Optional[str] = None # sticky session id PROXY_HOST = "gateway.example-proxy.com:7777" PROXY_USER = "YOUR_USER" PROXY_PASS = "YOUR_PASSWORD" def build_proxy_url(t: GeoTarget) -> str: # ADAPT THIS to your provider's documented syntax. parts = [PROXY_USER, f"country-{t.country.lower()}"] if t.city: parts.append(f"city-{t.city.lower()}") if t.session: parts.append(f"session-{t.session}") username = "-".join(parts) return f"http://{username}:{PROXY_PASS}@{PROXY_HOST}" Keep credentials in environment variables in real code, not in source.
If you don’t have a proxy provider yet, you can test the setup above with one.
Get started: Proxy-Seller, with referral code
HARSH15.
Step 1: Verify the exit location
Don’t assume the proxy put you where you asked. Ask a geo-IP service where the exit IP is, and stop early if it’s wrong.
import requests def check_exit(proxy_url: str) -> dict: proxies = {"http": proxy_url, "https": proxy_url} r = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=20) r.raise_for_status() return r.json() # includes ip, city, region, country, timezone def assert_geo(t: GeoTarget) -> dict: info = check_exit(build_proxy_url(t)) if info.get("country", "").upper() != t.country.upper(): raise RuntimeError(f"Wanted {t.country}, got {info.get('country')} ({info.get('ip')})") if t.city and info.get("city", "").lower() != t.city.lower(): # Don't hard-fail on city; log and decide per project. print(f"Warning: wanted {t.city}, geo-IP says {info.get('city')}") return info if __name__ == "__main__": print(assert_geo(GeoTarget("DE", "Berlin"))) Two things to keep in mind:
- A single geo-IP service is just one opinion, so for city-critical work you'll want to query two and compare.
- Free tiers are rate-limited, so cache the result for the duration of the session rather than making the call on every request.
Step 2: Make the request look consistent
A German exit IP sending Accept-Language: en-US is the kind of mismatch some sites notice, so set the header to match the location.
LANG_BY_COUNTRY = { "DE": "de-DE,de;q=0.9,en;q=0.5", "FR": "fr-FR,fr;q=0.9,en;q=0.5", "US": "en-US,en;q=0.9", "GB": "en-GB,en;q=0.9", "IN": "en-IN,en;q=0.9,hi;q=0.6", } def headers_for(country: str) -> dict: return { "User-Agent": ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/124.0 Safari/537.36" ), "Accept-Language": LANG_BY_COUNTRY.get(country.upper(), "en-US,en;q=0.9"), } def fetch(url: str, t: GeoTarget) -> requests.Response: proxy = build_proxy_url(t) return requests.get( url, headers=headers_for(t.country), proxies={"http": proxy, "https": proxy}, timeout=30, ) Swap the User-Agent for a current browser string. The one above is only an example.
Step 3: Sticky sessions or rotating IPs
Flows that are location-dependent (e.g., setting a ZIP, adding to cart, paginating) will break if the IP changes halfway through. Most providers will have sticky sessions, which will keep the same exit IP for a period of time, which is what the session field above is for.
- Use a sticky session in any scenario where there is state: login, cart, ZIP selection, multi-page pagination, etc.
- Rotate the IPs for the independent fetches (one product page per request essentially) so the load gets spread out.
import uuid def new_session_target(country: str, city: str | None = None) -> GeoTarget: return GeoTarget(country, city, session=uuid.uuid4().hex[:10]) Step 4: Scrape several locations at once
To compare one page across locations, run the requests concurrently with a bounded pool. This uses httpx, where proxy= is the current argument. Older tutorials show proxies=.
import asyncio import httpx TARGETS = [ GeoTarget("US", "newyork"), GeoTarget("GB", "london"), GeoTarget("DE", "berlin"), GeoTarget("IN", "mumbai"), ] async def fetch_geo(url: str, t: GeoTarget, sem: asyncio.Semaphore) -> tuple[GeoTarget, int, str]: async with sem: async with httpx.AsyncClient( proxy=build_proxy_url(t), headers=headers_for(t.country), timeout=30, follow_redirects=True, ) as client: r = await client.get(url) return t, r.status_code, r.text async def main(url: str): sem = asyncio.Semaphore(4) # keep concurrency modest results = await asyncio.gather(*(fetch_geo(url, t, sem) for t in TARGETS)) for t, status, body in results: print(t.country, t.city, status, len(body)) if __name__ == "__main__": asyncio.run(main("https://example.com/product/123")) Step 5: Browser scraping with matching locale and timezone
On JavaScript-heavy sites, the proxy alone isn’t enough, because the page can read your timezone, your locale, and, if allowed, your coordinates. Playwright lets you set all of these per context. One handy trick is reading the timezone from the exit-IP lookup, so the browser agrees with the proxy.
from urllib.parse import urlparse from playwright.sync_api import sync_playwright CITY_COORDS = { "berlin": {"latitude": 52.5200, "longitude": 13.4050}, "london": {"latitude": 51.5072, "longitude": -0.1276}, } def scrape_with_browser(url: str, t: GeoTarget, locale: str): info = assert_geo(t) # exit IP check from Step 1 tz = info.get("timezone", "UTC") p = urlparse(build_proxy_url(t)) # Playwright wants server/username/password separately with sync_playwright() as pw: browser = pw.chromium.launch(headless=True) context = browser.new_context( proxy={ "server": f"http://{p.hostname}:{p.port}", "username": p.username, "password": p.password, }, locale=locale, # e.g. "de-DE" timezone_id=tz, # e.g. "Europe/Berlin" geolocation=CITY_COORDS.get((t.city or "").lower()), permissions=["geolocation"] if t.city else [], ) page = context.new_page() page.goto(url, wait_until="domcontentloaded") html = page.content() browser.close() return html If your Playwright and Chromium versions are refusing per-context proxies, then set the proxy in launch() and run one browser per location. Only grant the geolocation permission when the site actually asks for it (granting it everywhere is itself a pattern).
Step 6: Catch silent geo failures
The most costly failure is a 200 response with the wrong content. Add a check for evidence of the right location.
import re def looks_like_expected_geo(html: str, country: str) -> bool: checks = { "DE": [r"€", r"\bMwSt\b|\binkl\. MwSt"], "GB": [r"£"], "US": [r"\$\d"], "IN": [r"₹"], } patterns = checks.get(country.upper(), []) return all(re.search(p, html) for p in patterns) if patterns else True A few other cheap signals to look for:
- The final URL after redirects, such as
/de/,.co.uk, or?country=DE. - A “Deliver to” label or another visible location marker on the page.
- The currency symbol or ISO code.
- A country-selection interstitial, which means the site couldn’t decide and gave you a default.
Log a sample of failures along with the exit IP, so you can tell whether the proxy, the target’s geo database, or your headers caused the mismatch.
Common problems and fixes
- If you're getting the US version even though you're using a foreign proxy, the site is almost certainly reading a cookie from a previous visit, or you've stumbled upon a cached response. Use a fresh context/cookie jar for each location and look out for geo query params.
- If the country is correct and the city is wrong, that can happen. Check what the page itself says rather than just the ipinfo, then try another IP from the pool and fall back to the state level.
- If you're seeing more CAPTCHAs on residential IPs from a single city, then you've got a small pool for that particular location being heavily reused. Rotate within the location, slow down, and cut parallelism per city.
- If the content is different in your browser than it is in your scraper at the same location, then compare the headers, cookies, and the rendered DOM. A different locale/timezone is a very common cause, and so is a missing consent cookie.
- If prices flip between requests, the site may be A/B testing prices regardless of location. Repeat each geo several times and compare the distributions before blaming location.
Cost and efficiency
- Residential bandwidth is normally the expensive part, so block out images, fonts, and media in browser contexts if all you really need is text and prices.
- If the geo-specific data is loaded from a small JSON endpoint, then just call that endpoint directly rather than rendering the entire page.
- Verify the exit IP once per session rather than once per request.
- Track the success rate per location. A location that's suddenly failed is much more likely to be a pool issue than a code issue.
Legal and ethical checks
A proxy changes where you're from. It doesn’t in any way change what you’re permitted to do.
- Read the target’s terms of service and
robots.txt, and respect rate limits. - Don’t collect any personal data that you have no basis to process, as privacy laws such as GDPR will apply to the data you collect wherever your proxy is sitting.
- Pick a provider that is open about how its residential IPs are sourced and whether the device owners consented.
- Don’t use geo-spoofing to get around access controls on content you aren’t licensed to see.
- For a specific project you’re unsure about, get legal advice instead of relying on an article.
Quick checklist
- Use the narrowest targeting you can get away with: country first, city only if the content requires it.
- Keep provider syntax in one function.
- Check the country (and city) of the exit IP prior to you starting the scraping.
- Match
Accept-Language, locale, and timezone to the exit location. - Use sticky sessions for stateful flows and rotation for independent fetches.
- Check that the response really reflects the target location.
- Fall back from city to region to country, and log which level you got.
- Track the success rate per location.
Frequently Asked Questions (FAQs)
What is geo-targeted scraping?
Geo-targeting web scraping is the process of extracting information from websites as if browsing from a specific geographic location, such as a city or country, usually using a proxy server in that location. Websites charge different prices and display different inventories, search results, and languages according to the browser's geographic location.
What's the difference between country-level and city-level proxy targeting?
Country-level targeting gets you an exit IP somewhere in the country you want. Geo IP databases pretty much always agree on this. City-level targeting gets you an IP in a specific city, and that's far less reliable, as the databases often disagree on which city an IP actually belongs to. If all that changes by country is currency, language, and catalog, then you're fine at country level. Use city level if you're looking at local content (local search results, local store inventory, etc.) and check exactly what the target is showing you.
Do I need residential proxies for geo-targeted scraping?
Not necessarily. Datacentre proxies are cheaper and faster, and suit country-level scraping on sites with minimal bot protection. Residential proxies are the way to go if you're doing city-level work, local SERPs, and on sites with a lot of bot protection, as they come from a much wider range of consumer ISPs. ISP (static residential) proxies are in the middle ground if you're after stable IPs with a good reputation.
How can I check that my proxy is in the location I requested?
Send a request through the proxy to a geo-IP service such as ipinfo.io/json and compare the returned country, city, and timezone with what you asked for. Do it once per session and stop on a country mismatch. For city-critical work, ask a second service as well, and look at what the target page reports (a “Deliver to” label, the currency, the redirect path), since the target may use a different database.
Should I go for sticky sessions or rotating IPs?
Use sticky sessions when you have a flow that has state (e.g., setting a ZIP, logging in, adding to cart, paginating), as an IP change in the middle of that flow can reset the location or even result in a block. Use rotating IPs for independent single-page fetches (i.e., one product page per request) so the load is distributed across the pool.
Should I go for sticky sessions or rotating IPs?
Use sticky sessions when you've got a flow that has state (e.g., setting a ZIP, logging in, adding to cart, paginating), as an IP change in the middle of that flow can reset the location or indeed cause a block. Use rotating IPs for independent single-page fetches (i.e., one product page per request), so the load gets spread across the pool.
