I ran a scraping platform that processed millions of pages a day at roughly 95% extraction success, around three seconds per page. The fetch-and-parse code, the part everyone thinks of as "the scraper", was a tiny slice of the whole thing. The years went into everything around it.
Here's what actually broke, more or less in the order it broke.
The queue with no manners
Crawling is bursty in a way that surprises you the first time. One category page fans out into a few hundred product URLs. A sitemap refresh dumps half a million at once. Meanwhile your parsers chew through pages at a steady rate that doesn't care about your ambitions.
Our first queue was effectively unbounded. It absorbed every burst happily, Redis memory climbed for two days, and then the whole thing fell over at once instead of slowing down gracefully. Lesson: if your queue can't say no, it's not a queue, it's a landfill. Bound it, and make producers block or shed when it's full. A crawler that pauses discovery for an hour is a non-event. A crawler that OOMs the broker is a weekend.
Retries that stampede
A target site starts throwing 500s for two minutes. Fine, that happens. Every failed page gets rescheduled with the same fixed backoff. Which means 40,000 pages land back on that site in the same ten-second window, it tips over again, and now their WAF has opinions about you. You caused the second outage yourself.
Two fixes, both mandatory: full jitter on backoff, and a per-domain circuit breaker so a struggling site stops receiving traffic entirely for a while.
import random
def backoff(attempt, base=2.0, cap=300.0):
# full jitter: spread retries across the whole window
return random.uniform(0, min(cap, base * 2 ** attempt))
That one-liner is the difference between "we retry politely" and "we DDoS people by accident".
Dedupe before fetch, not after
The same product reaches you through six URLs. Tracking params, color variant params, three different category paths, a mobile subdomain. If you dedupe after fetching, you paid for six fetches to keep one page. At millions of pages a day that's real money and real block-risk for zero data.
Canonicalize at enqueue time. Strip the junk params, pick one canonical form, check the seen-set before the URL ever enters the queue. And watch that seen-set: it grows forever unless you give it a TTL or accept a small false-positive rate with something bloom-filter-shaped.
Parser drift, the quiet one
Sites almost never break your parser loudly. They drip. A price moves from a DOM node into an embedded script blob. An image attribute gets renamed. Your extraction still "succeeds", the run is green, and one field quietly goes null on 30% of pages. Nobody notices until a downstream consumer asks why half the prices vanished last Tuesday.
Field-level fill rates saved us. For every field, track what fraction of pages yielded a value, keep a rolling baseline per site, alert on the delta. A green run only proves the code didn't crash.
The other thing that got us to 95%: never rely on a single extraction strategy. Embedded JSON, microdata, plain DOM selectors, run more than one and merge by confidence. When a site redesign kills one of them, the others carry it while you fix things, instead of the number going to zero overnight.
The CPU step hiding in an I/O system
Scraping feels like an I/O problem. Fetch, wait, fetch, wait. So you build it all
async and assume the network is the only thing you're waiting on. Then you profile a
slow day and find the bottleneck is HTML parsing.
Loading a big product page into an HTML parser and running extractors over it is
CPU-bound and synchronous. On a single-threaded runtime, every parse blocks the event
loop, which means it blocks the very downloads you thought were the slow part. Your
concurrency number looks healthy and your throughput doesn't match it, because workers
are queued behind each other's parsing, not behind the network.
The fix is to move the CPU-heavy parse off the main thread into a worker pool, and fall
back to inline parsing when the pool is disabled or a job errors. Downloads keep flowing
while parsing happens elsewhere. Obvious in hindsight, invisible until you measure it.
It ended up one of the bigger single throughput wins I got at scale.
You will need to debug the past
The nastiest questions arrive late. "Why did brand X prices go missing on the 14th?" It's the 19th. The pages have changed since. Your logs say everything was fine.
We got out of that hole with a typed per-page event stream plus archived raw HTML. Every page emitted structured events (fetched, parsed, extracted, with typed payloads) and the raw response got stored. So "what happened on the 14th" became a replay: pull the archived pages, re-run the current parser against them, diff the output. Debugging in the past tense. Without replay you're reconstructing incidents from vibes.
The uncomfortable summary
None of this is glamorous. There's no clever algorithm in this post, and that's the point. At small scale, scraping is a parsing problem. At millions of pages a day it's a distributed systems problem wearing a parsing costume, and the failure modes are the boring ones: backpressure, thundering herds, silent data decay, no audit trail.
All of this is what I'm productizing at Cartpie, e-commerce product data as an API so you don't have to own any of the machinery above. Free tier is live.
