Every scraping tutorial says "use proxies." Almost none of them explain the one decision that actually determines whether your scraper survives: sticky or rotating?
Pick wrong in either direction and you get the two classic failure modes:
- Rotating when the site expects session consistency → logged out mid-flow, cart emptied, "suspicious activity" emails
- Sticky when the site counts requests per IP → rate limited into oblivion after 50 requests
Let's make sure you never hit either.
Rotating proxies: the crowd
Each request exits from a different IP in the pool. To the target site, your traffic looks like many unrelated visitors instead of one very busy one.
# Most gateways rotate by default — one line:
proxies = {"https": f"http://{USER}:{PASS}@gate.thordata.com:9000"}
for url in urls:
r = requests.get(url, proxies=proxies, timeout=30)
# every request = a fresh IP
Use rotating when:
- Crawling many public pages (product catalogs, article archives)
- SERP tracking and price monitoring across markets
- Any job where no request depends on the previous one
The mental model: rotation is for when the site counts requests per IP.
Sticky sessions: the regular
A sticky session pins you to one residential IP for a window (usually up to 30 minutes). Now your scraper behaves like one consistent person.
# Same gateway, session ID appended to the username:
sid = uuid.uuid4().hex[:10]
user = f"{USER}-session-{sid}"
proxies = {"https": f"http://{user}:{PASS}@gate.thordata.com:9000"}
# login → browse → paginate, all from the SAME IP
s = requests.Session()
s.proxies = proxies
s.post("https://example.com/login", data=creds)
s.get("https://example.com/dashboard?page=2") # still logged in ✅
Use sticky when:
- Anything behind a login
- Multi-step flows: checkout, booking, forms
- Paginated dashboards where the session must survive
- Account management (one sticky IP per account = looks like one household)
The mental model: stickiness is for when the site remembers who you are.
The decision table
| Your task | Choice |
|---|---|
| Crawl 10k public pages | Rotating + throttling |
| Price monitor across countries | Rotating + geo-targeting |
| Scrape behind login | Sticky |
| Manage 50 accounts | Sticky, one IP each |
| Login → export report | Sticky, then release |
The advanced pattern: rotate the sessions, not the requests
The best production scrapers use both. Divide your work into "identities": each identity gets a sticky session, does a human-sized chunk of work (5–20 requests), then you release it and spin up a fresh one.
def scrape_as_identities(urls, chunk_size=10):
for i in range(0, len(urls), chunk_size):
sid = uuid.uuid4().hex[:10] # new identity
s = session_for(sid) # new sticky IP
for url in urls[i:i+chunk_size]:
yield fetch(s, url)
time.sleep(random.uniform(3, 8)) # human-ish gap
This is the pattern that survives: no single IP accumulates volume, and every sub-flow is internally consistent.
Quick FAQ
"My sticky session died after 30 minutes!" — That's the window expiring. Design your jobs to finish within it, or re-login under a new session ID.
"Can I get the same IP again tomorrow?" — Generally no; treat session IDs as ephemeral identities, not permanent addresses. If you need permanent IPs, that's what static residential IPs are for (Thordata sells them at $0.75/IP — one stable IP you keep, ideal for account-holding workloads).
"Does rotation slow me down?" — Negligibly. The pool lookup is fast; your bottleneck should be your politeness delays, not the proxy.
Try it yourself
I use Thordata in all my examples — their free trial covers enough traffic to test both patterns end-to-end, rotating and sticky are both supported, and code thor020 gets you 10% off if you continue. (Disclosure: that's my referral link.)
Next in this series: geo-targeted scraping — why the same URL returns different content per country, and how to capture each version. Follow if that's your thing.
