I spent a good week convinced my SERP scraper was buggy. The pipeline asked Google for "best running shoes" with gl=de and hl=de, expecting German results, and got back a page that looked unmistakably American: prices in USD, retailers I had never heard of shipping to New Jersey, and a sticky "Did you mean" in English. I triple-checked the parameters. They were right there in the URL. I checked my parsing. Fine. It took me until I ran the exact same query through a box in Frankfurt to realize the parameters were the least important part of the request. Google had already decided where I was, and it had not decided it was Frankfurt.
This is the part of search scraping that does not fit in a tutorial. The tutorials tell you to set gl (the country you want) and hl (the language you want the interface in), and then they quietly assume you are hitting Google from somewhere neutral. You are not. You are hitting it from an IP, and the IP has a geography, and the geography is a signal that can override, blend with, or blunt your declared parameters. If you are building anything that depends on what a localized SERP actually contains - price monitoring, SEO rank tracking, ad verification, market research - then which proxy you exit from is not a plumbing detail. It is most of the product.
What Google actually keys off
There are at least five signals that shape the result set, and they do not have equal weight:
- Geolocation of the exit IP. Not the country, ideally the city or region. A residential or mobile IP resolves to a metro area. A datacenter IP resolves to whatever the block's registry says, which is often a datacenter campus in a different city than the one you "chose."
-
The top-level domain and redirect. Hitting
google.comfrom a German IP tends to bounce you togoogle.de. Hittinggoogle.dedirectly from a US IP still works but carries a mismatch between where you claim to be and where you are. -
glandhl. These are hints, not commands. They tune personalization and interface language. They do not force the local pack, and they absolutely do not move you across a hard geolocation boundary. - Cookie and account personalization. If you send a consent cookie tied to a profile, or you are logged in, history can dominate everything else. Fresh sessions matter.
- The consent interstitial itself. In the EU especially, a first visit from certain IPs gets a consent wall before any results render. Datacenter ranges trip this constantly. Your scraper reads a 200 with a "before you continue" page and happily scrapes zero results, then reports rank 99 for a query that has no rank.
Notice that the parameter you would reach for first, gl, is near the bottom of that list, and the one nobody puts in a tutorial - the exit IP - is at the top. That inversion is where most DIY SERP pipelines quietly rot.
The comparison I keep having to redo
Three ways to get a localized SERP, roughly in order of how much control you keep:
- Datacenter proxy + raw query. Cheapest, fastest, and the least honest about geography. The ASN is obviously a datacenter, Google degrades the localization, and you get the consent wall or a US-flavored result. Fine for a quick sanity check. Not fine for anything you will defend in a meeting.
-
Residential or ISP proxy + raw query. The exit IP carries real metro-level geolocation, so
gland the IP agree, the local pack behaves, and the consent wall is largely gone. You are now parsing HTML yourself and rebuilding structure on every SERP layout change. - A managed SERP API. Someone else maintains the geo routing, the parsing, and the breakage. You trade control and per-query economics for not having this job at all.
I default to a residential or ISP pool for anything I own end to end, because the thing I am actually buying is geographic truth, and geographic truth is a property of the IP, not of the URL.
The check that found my bug
Here is the small harness I now run before I trust any SERP pipeline. It fires the same query through proxies pinned to two different countries, and diffs the result. It is plain Python with httpx. The proxy strings are placeholders - swap in your own host:port with username:password, or your provider's session parameters.
import httpx
from html.parser import HTMLParser
import re
QUERY = "best running shoes"
# Each entry is (label, httpx proxy url, Google domain, gl, hl)
TARGETS = [
("US", "http://USER:PASS@gw-us.example.com:8000", "google.com", "us", "en"),
("DE", "http://USER:PASS@gw-de.example.com:8000", "google.de", "de", "de"),
("JP", "http://USER:PASS@gw-jp.example.com:8000", "google.co.jp","jp", "ja"),
]
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126 Safari/537.36")
def probe(label, proxy, domain, gl, hl):
url = f"https://{domain}/search"
params = {"q": QUERY, "gl": gl, "hl": hl, "num": "20"}
headers = {
"User-Agent": UA,
# A fresh, non-personalized session. No cookie jar reuse across probes.
"Accept-Language": f"{hl},en;q=0.7",
}
with httpx.Client(proxy=proxy, headers=headers, timeout=20.0,
follow_redirects=True) as client:
r = client.get(url, params=params)
# 1. What geography does the exit IP actually claim?
try:
ipinfo = client.get("http://ip-api.com/json/fields=country,city,query,as")
loc = ipinfo.json()
except Exception as e:
loc = {"error": str(e)}
body = r.text
# 2. Did we get results or a consent / no-results page?
consent = ("consent.google" in str(r.url)) or ("before you continue" in body.lower())
positions = re.findall(r'<h3[^>]*>(.*?)</h3>', body)
titles = [re.sub(r'<[^>]+>', '', p).strip() for p in positions][:5]
return {
"label": label,
"final_url": str(r.url),
"status": r.status_code,
"exit_loc": loc,
"consent_wall": consent,
"top5": titles,
}
for t in TARGETS:
row = probe(*t)
print(f"\n== {row['label']} ==")
print("final url :", row["final_url"])
print("http status :", row["status"])
print("exit geo/ip :", row["exit_loc"])
print("consent wall :", row["consent_wall"])
print("top results :", row["top5"])
Two things this surfaces. First, the exit_loc line: if you requested gl=de but the IP resolves to California, your result diff is now explained before you even parse a thing. Second, consent_wall: a green HTTP 200 with a consent page is a false success, and it is the single most common reason a "rank tracker" reports that a brand simply vanished from page one in Germany. It did not vanish. You never got a German page.
Same idea, one line, if you just want a smoke test from the terminal:
# What does Google think my city is, versus what I asked for?
curl -s -x http://USER:PASS@gw-de.example.com:8000 \
"http://ip-api.com/json/fields=country,regionName,city,query"
Results, and the traps
Running that harness against a datacenter box and a residential box in the same country, the honest findings were:
- The datacenter IP was geocoded to the wrong metro. It said the country was right and the city was wrong. Google's local pack keyed off the wrong city and my "rank in Munich" tracking was actually rank in some IRS-building suburb. The result list was not empty; it was confidently, usefully wrong, which is worse than empty.
-
gldid not fix it. Settinggl=dewith a US-resolving datacenter IP produced a US result set a large fraction of the time. The parameter is a tiebreaker, not a teleport. -
The consent wall appeared on the first request from datacenter ranges roughly enough that I stopped treating a fresh session as a nicety and started treating it as a requirement. Randomize, don't reuse; clear cookies; handle the redirect explicitly instead of
follow_redirects=Trueswallowing it into a "no results" page. -
Residential and static ISP pools agreed with
gl. Once the IP and the declared country matched, the local pack behaved, the currency localized, and the diff between my two countries looked like two real SERPs instead of one real SERP and a shrug.
The trap I still fall into occasionally: language versus region. hl is interface language; gl is region; a person can want Japanese interface, US region, English results. Treat them as three independent knobs and always log the exit geography, because it is the fourth knob you did not set.
My rough rule of thumb now, and the only number I checked live today rather than from memory: Thordata's residential pool starts from $0.65/GB as of this writing (list $1.05/GB), and for pure SERP geo-truth that is the line I reach for first. ISP and datacenter tiers are priced differently per IP and per GB, so I would not paste a figure I have not just looked at - go read the pricing page before you quote it in a design doc, same as I did.
Disclosure, plainly: I work with Thordata, which sells the residential and ISP pools I lean on for geo-accurate scraping. The diff harness above is generic; run it against whatever proxy you already have, and let it tell you the uncomfortable truth about your own exit IP before you buy anything from anyone, including us.
