I Tested 5 Proxy Types for Web Scraping — Here's What Actually Matters

Last month I rebuilt a scraping pipeline that had been limping along on a bargain-bin datacenter proxy plan. The symptoms were familiar to anyone who's done this at any scale: success rate hovering around 60%, CAPTCHAs on every third session, and a monthly bill that made no sense relative to the data we were actually collecting.

So I did what any reasonable developer would do: I spent two weeks testing every proxy type I could get my hands on against the same set of targets, with the same code, and wrote down what actually happened.

This post is the write-up. No affiliate nonsense, no "top 10 providers" list — just what each proxy type is, when it works, when it fails, and the code patterns that matter regardless of which type you pick.

The Setup

Test workload: product pages from a handful of mid-size e-commerce sites, a couple of search-heavy directories, and one notoriously aggressive social platform. Roughly 50k requests per proxy type, spread across a week, using Python + requests. Same headers, same throttling (2–4s between requests per host), same retry logic. The only variable was the proxy layer.

Here's the baseline code everything else builds on:

import requests

PROXY = "http://user:pass@proxy.example.com:8080"

proxies = {
    "http": PROXY,
    "https": PROXY,
}

def fetch(url: str) -> requests.Response:
    return requests.get(
        url,
        proxies=proxies,
        timeout=30,
        headers={
            "User-Agent": (
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/128.0.0.0 Safari/537.36"
            ),
            "Accept-Language": "en-US,en;q=0.9",
        },
    )

If you're copy-pasting proxy examples from blog posts, stop here and internalize one thing: most scraping failures are not proxy failures. They're fingerprint failures, rate failures, or session failures that a better proxy happens to mask. Keep that in mind as we go through the types, because the marketing for each type will happily take credit for all three.

1. Datacenter Proxies

What they are: IPs allocated to hosting providers and cloud ranges (AWS, Hetzner, OVH, and thousands of smaller hosts). Cheap, fast, and trivially identifiable — the ASN alone gives them away.

Results: ~55–65% success on the e-commerce targets, near-zero on the social platform. Latency was excellent (often under 300ms to first byte). Cost per GB was the lowest of anything tested by a wide margin.

When to use them:

• Targets with weak or no bot protection (public APIs, small sites, RSS-like endpoints)
• High-volume, low-value requests where a 40% failure rate is acceptable because retries are cheap
• Geo-checking and uptime monitoring, where you just need an IP in a country, not a trustworthy one

When to avoid: Anything behind Cloudflare Bot Management, DataDome, PerimeterX, or Akamai in a strict configuration. You'll burn your retry budget and learn nothing.

2. Residential Proxies (Rotating)

What they are: IPs belonging to real consumer ISPs, routed through devices on home connections. Sold by the GB, usually with per-request or timed rotation.

Results: ~92–96% success on e-commerce, ~80% on the social platform with careful session handling. Latency varied wildly — 800ms on a good exit node, 4s+ on a bad one. This variance is the price of admission.

The single biggest improvement in my whole test came from rotation strategy, not from switching providers. Two patterns matter:

Per-request rotation — fresh IP every request. Good for stateless, high-parallelism workloads:

import random
import string

def session_id() -> str:
    return "".join(random.choices(string.ascii_lowercase + string.digits, k=8))

def rotating_proxy() -> dict:
    # Most rotating residential gateways encode the session in the username.
    # A new session ID = a new exit IP.
    user = f"customer-{session_id()}"
    proxy = f"http://{user}:password@gateway.example.com:8080"
    return {"http": proxy, "https": proxy}

Sticky sessions — same IP for N minutes. Mandatory for anything with login state, carts, or multi-step flows:

class StickySession:
    """One exit IP per logical task; rotate only when the task ends or fails."""

    def __init__(self, ttl_requests: int = 50):
        self.ttl = ttl_requests
        self.count = 0
        self.session = self._new_session()

    def _new_session(self) -> requests.Session:
        s = requests.Session()
        sid = session_id()
        proxy = f"http://customer-{sid}:password@gateway.example.com:8080"
        s.proxies = {"http": proxy, "https": proxy}
        return s

    def get(self, url: str, **kw) -> requests.Response:
        if self.count >= self.ttl:
            self.session = self._new_session()
            self.count = 0
        self.count += 1
        return self.session.get(url, timeout=30, **kw)

Honest trade-off: you're paying per GB for traffic that transits someone's home connection. That makes residential the wrong tool for downloading large payloads, and the right tool when the cost of a blocked request (a missed price change, a dead account) exceeds the cost of the bandwidth.

3. Mobile Proxies

What they are: IPs from carrier networks (4G/5G). The magic here is CGNAT: hundreds or thousands of real subscribers share a single public IP, so aggressive blocking hurts the site's real users — which means sites are very reluctant to block mobile IPs outright.

Results: Highest success rate of the test (~97% even on the social platform). Also the slowest and most expensive per GB, typically 3–10× residential pricing.

When to use them: Social platforms, account-adjacent automation, anywhere a block is catastrophic rather than merely annoying.

When to avoid: Bulk scraping. You'll go bankrupt feeding a price monitor through mobile bandwidth. Also note that mobile IPs rotate on their own schedule (tower handoffs, NAT timeouts), so long sticky sessions are not guaranteed no matter what the sales page says.

4. ISP Proxies

What they are: Datacenter-hosted IPs that are registered to consumer ISPs. You get the speed and stability of a datacenter connection with an ASN that looks residential. Typically sold per-IP, per-month rather than per-GB.

Results: ~90% success on e-commerce, stable latency (~400ms), and — uniquely in this test — the same IPs stayed fast and unblocked for the entire week. No rotation lottery.

When to use them: Long-running jobs against a fixed set of targets, monitoring tasks, anything where you want residential-grade trust with datacenter-grade predictability and a flat monthly bill.

When to avoid: Workloads that need thousands of distinct IPs. Per-IP pricing means breadth costs money, and if you hammer one ISP IP hard enough, it will get flagged eventually — the pool you can fall back to is much smaller than a rotating residential pool.

5. Static (Dedicated) Proxies

A quick taxonomy note, because vendors abuse these terms: "static" describes behavior (the IP doesn't change), not a network type. A static proxy can be a datacenter IP or an ISP IP that you keep for the duration of your plan. In my test setup, static ISP was the workhorse configuration for the monitoring jobs — one IP per target site, requests spread politely, zero surprises across two weeks.

The general rule: rotating for breadth, static for depth. If a job revisits the same target repeatedly and failure has consequences, static wins. If a job fans out across many targets and individual failures are cheap, rotate.

Detecting Blocks (and Not Lying to Yourself About It)

The most dangerous scraping system is one that reports 100% success while quietly ingesting garbage. Sites defend themselves on a spectrum:

1. Hard blocks: 403, 429, connection resets. Easy to detect, easy to handle.
2. Soft blocks: HTTP 200 with a CAPTCHA page, a "verify you're human" interstitial, or an empty-but-valid HTML shell. This is where pipelines go to die. Your parser returns empty results and nobody notices for a week.
3. Honeypots and poisoned data: 200 with subtly wrong data — prices off by a digit, shuffled listings, fake out-of-stock flags. Rare, but they exist specifically to corrupt datasets like yours.

Detection needs to be structural, not status-code-based:

BLOCK_MARKERS = (
    "captcha", "verify you are human", "access denied",
    "are you a robot", "unusual traffic", "px-captcha",
)

def classify(resp: requests.Response, expected_selector_found: bool) -> str:
    if resp.status_code in (403, 429, 503):
        return "hard_block"
    body = resp.text.lower()
    if any(m in body for m in BLOCK_MARKERS):
        return "soft_block"
    if resp.status_code == 200 and not expected_selector_found:
        return "suspect"  # 200 but the data isn't there — treat as failure
    return "ok"

Track these categories separately in your metrics. If your dashboard only shows HTTP status codes, you're flying blind.

And the retry logic that ties it together — retry on a different IP for hard/soft blocks, back off on 429s, and never retry a poisoned response:

import time

def fetch_with_recovery(url: str, task: StickySession, attempts: int = 4):
    for i in range(attempts):
        try:
            resp = task.get(url)
        except requests.RequestException:
            resp = None

        if resp is not None:
            verdict = classify(resp, expected_selector_found=has_data(resp))
            if verdict == "ok":
                return resp
            if verdict == "suspect":
                # Possible honeypot — log it, don't ingest, don't blindly retry
                raise SuspiciousResponse(url)
        # Hard block, soft block, or network error: new exit IP, then back off
        task.session = task._new_session()
        task.count = 0
        time.sleep(min(2 ** i, 30))
    raise BlockedError(f"{url} failed after {attempts} attempts")

Cost Optimization: Stop Burning Residential GB on JPEGs

Per-GB pricing means every byte counts, and the default behavior of most scraping stacks is a bonfire. In my first pipeline version, over 60% of residential traffic was images, fonts, and CSS — assets I was never going to parse.

The fixes, in order of impact:

1. Block static assets. If you're parsing HTML, you need HTML. In requests this means not following asset URLs at all; in browser automation (Playwright/Selenium), intercept and abort image/font/media requests. This alone halved my bill.
2. Route by target hardness. Send the easy 80% of targets through datacenter proxies at a fraction of the cost, and reserve residential for the hosts that actually block you. A simple per-host success-rate table in your code can automate this routing after a day of data.
3. Cache aggressively at the URL level. Product pages rarely change minute-to-minute. A 15-minute in-memory cache on listing pages eliminated a third of my requests entirely.
4. Compress and shrink. Send Accept-Encoding: gzip, br explicitly (some proxy gateways strip it), and use HEAD/conditional requests where the target supports ETags.
5. Right-size sessions. Sticky sessions that live too long waste the "freshness" you're paying for; sessions that rotate too often waste connection setup and look bot-like. 10–30 minutes is the sweet spot for most of my workloads.

After all five, the same crawl that cost me a painful amount in week one ran at roughly one third of the cost in week two — with a higher success rate, because blocked-target routing meant residential IPs were only spent where they mattered.

The Summary Table

What I'd Do Differently

Two lessons stuck. First, log the block taxonomy from day one — retrofitting detection onto a pipeline that assumed 200 = success cost me more time than the proxy testing itself. Second, match the proxy to the failure cost, not to the target's reputation. "This site is hard, use mobile" is worse advice than "this request is expensive to lose, use mobile."

For the residential and static ISP portions of this test I used IPStation ($2.50/GB dynamic residential, $3.99/month static ISP, and the 200MB free trial was enough to validate the setup before spending anything) — but honestly, every pattern in this post is provider-agnostic. Pick any reputable vendor, steal the detection and routing code, and you'll get 90% of the benefit.

The proxies were never the hard part. Knowing why a request failed — and what that failure should make your code do next — is the whole game.

Questions or better war stories? Drop them in the comments — especially if you've caught a target serving poisoned data in the wild. I want to hear how you detected it.