The scenario nobody warns you about
You re-crawl a few tens of thousands of pages on a schedule - price pages, listing pages, catalog metadata. Most of them are unchanged from run to run, so you did the right engineering thing: store the validators, send conditional GETs, enjoy the 304 Not Modified responses with their zero-byte bodies. Revalidation should feel nearly free.
And yet the job on the rotating pool did not get proportionally faster, and the cost model did not drop as much as the 304 ratio promised. Here is the uncomfortable mechanics: a 304 saves body bytes and nothing else. The DNS lookup (unless cached), the TCP connect, the TLS handshake, a full round trip from wherever your current exit happens to live - you pay all of it, per revalidation, to fetch an empty response. And a rotating proxy pool quietly destroys the one mechanism that makes revalidation cheap: connection reuse. The cache layer assumes connections persist. Rotation is designed around them not persisting.
I measured this on a live target this morning, and the gap between "revalidation is cheap" and "revalidation is cheap through a warm connection" turned out to be a factor of three to seven. This post walks through the mechanics, the numbers, the way edge caches muddy the picture, and a 25-line probe that tells you which regime each of your targets lives in.
What a conditional GET actually does
Two halves that people routinely conflate.
The first half is freshness. RFC 9111 says a cached response is fresh until its max-age window passes. While fresh, a correct client should not touch the network at all. When I fetched thordata.com's homepage today it came back with Cache-Control: max-age=604800 (a week) and Age: 488. A browser that loaded it this morning will not revalidate it all week. Most scrapers ignore Cache-Control and revalidate everything, on every run, which is a legitimate policy choice - your crawl schedule already IS your freshness policy - but it's worth knowing that this is the choice you're making.
The second half is revalidation. Once the entry is stale, the client asks with a validator: If-None-Match for the ETag, If-Modified-Since for the Last-Modified stamp. The server either returns 304 with headers only, or a fresh 200 with the body. Note what a 304 is not: it is not a cheaper response class. It is a full HTTP exchange that happens to omit the body. The transport cost is identical to the cost of a 200 minus the bytes streamed.
So the price of a 304 is: connect cost + round-trip + server time. The moment you put a proxy in the path, "connect cost" means connect to the gateway, tunnel to the target through it, TLS through the tunnel - per connection.
Where the pool breaks it
Keep-alive amortizes the connect cost across every request on the connection. HTTP/1.1 with a warm pool makes revalidations dirt cheap - first request pays the handshake, requests two through N pay round-trip only. Proxy rotation inverts that economics deliberately: every new exit means the old tunnel is gone, and the next request starts the TLS ladder again from scratch.
Here is what that looks like measured, same origin (www.thordata.com), curl 8.17.0 on a Schannel build, 2026-09-24, one workstation, direct route:
cold connect, conditional GET, right validator -> 304, 0 bytes, 0.83 s total, 0.62 s in TLS handshake
cold connect, same thing again (run variance) -> 304, 0 bytes, 1.47 s total, 1.25 s in TLS handshake
same connection, transfers #2 and #3 -> 304, 0 bytes, 0.22 s / 0.21 s total, no new connect (num_connects=0)
cold connect, full GET -> 200, 1,361,071 bytes, 3.00 s
cold connect, validator mismatch (wrong date) -> 200, 1,361,071 bytes, 1.99 s
Read the first three lines together. The same 0-byte response cost 0.8-1.5 seconds when it needed a fresh TLS handshake, and 0.21 seconds when it rode a warm connection. The handshake is not a rounding error on a 304 - it was up to 85% of the total time of the request. Rotation buys you exit diversity and target-side session freshness; what it costs, invisibly, is that every revalidation you do now pays the connection tax a warm client never pays. (If you have already profiled rotation's per-connection TLS tax as a throughput problem, good - this post is the adjacent decision nobody profiles: the cache-policy question of whether you should send the conditional request at all, from a cold connection, on a budget that also gates your 200s.)
And then there's the billing interaction, which cuts the other way depending on your line. If you pay per GB - the residential line at Thordata starts from $0.65/GB (struck-through $1.05/GB; re-verified on the pricing page today) - then 304s genuinely save money no matter which connection regime you're in: 40,000 revalidations of that same 1.3 MB page avoid roughly 52 GB of tunnel traffic. The bytes saved are real dollars; nobody disputes this part. But if your bill is per-IP flat - the datacenter line shows from $0.75/IP (struck-through $1.50/IP on the same page today) - then bandwidth is not your constraint at all. Your constraints are wall-clock crawl duration and per-target request budget, and in that world, revalidating through rotated exits spends the same request budget at 3-7x the latency. The 304 looks like a win on the response log and is a wash or a loss on the stopwatch.
The edge-cache wrinkle: your validator was issued by some server
Validators are generated by whatever answered you. Behind a CDN - and both example pages above are edge-served; my fetches showed cf-cache-status: HIT with a growing Age - two exits may land on two PoPs whose copies were validated against origin at different moments, whose weak ETags are edge-generated, or whose Last-Modified granularity differs. Through exit A you capture W/"abc"; through exit B, the same bytes come back as a 200 because B's edge never had your conditional request's referent cached. Or the inverse: B answers 304 from its own context, which is strictly evidence about B's view of the resource, not a global statement that the page did not change.
There is a second, quieter version of this in your own data model: a 200 fetched from a US exit and a 304 confirmed from an EU exit are statements about different served variants for geo-personalized pages (currency, availability, price). If the goal is a consistent dataset, the cache key needs the exit's context in it, not just the URL.
I want to be straight about evidence here: single route, single workstation, one origin. Today I could not reproduce cross-PoP validator variance because my probe path never left my own egress - dev.to's own API kept timing out while I tried to gather a second measurement (transient 000s for a good chunk of this morning, honest flakiness, no data). Treat the multi-edge section as protocol reasoning plus one testable prediction: if you run the probe below through two different exits and the 200-bodies get different digests or the same validator behaves differently, you have caught it live.
A probe for which regime your targets are in
"""cache_regime_probe.py - pip install requests
Run twice per target URL:
1) direct (proxies=None) -> warm-connection regime
2) through your rotating gateway -> cold-connection regime
Compare per-request `sec` and bytes to price your rotation.
"""
import hashlib, json, sys, time
import requests
def probe(url, proxies=None, samples=6):
s = requests.Session()
rows, validator = [], None
for i in range(samples):
headers = {}
if validator:
kind, val = validator
headers = {"If-None-Match": val} if kind == "etag" else {"If-Modified-Since": val}
t0 = time.monotonic()
try:
r = s.get(url, headers=headers, proxies=proxies, timeout=(5, 15))
except requests.RequestException as e:
rows.append({"n": i, "outcome": "net_error", "err": type(e).__name__})
continue
dt = time.monotonic() - t0
if r.status_code == 304:
outcome = "304"
elif r.status_code == 200:
outcome = "200"
etag, lm = r.headers.get("ETag"), r.headers.get("Last-Modified")
validator = ("etag", etag) if etag else (("lm", lm) if lm else None)
else:
outcome = str(r.status_code)
rows.append({
"n": i, "outcome": outcome, "bytes": len(r.content),
"sec": round(dt, 3),
"digest": hashlib.sha256(r.content).hexdigest()[:12] if outcome == "200" else None,
"age": r.headers.get("Age"), "edge": r.headers.get("cf-cache-status"),
})
if i == samples // 2:
s.close(); s = requests.Session() # tear the connection down once, mid-run
return rows
if __name__ == "__main__":
url = sys.argv[1] if len(sys.argv) > 1 else "https://www.thordata.com/"
print(json.dumps(probe(url), indent=2))
The mid-run Session teardown is the whole point: it separates each run's first-half numbers (warm) from its second half (cold-connect), on the same route, same minute. Through a per-request rotating gateway, every request behaves like the second half. Through a sticky session, you can keep most revalidations in the warm regime - at the cost of concentrating all those requests on one identity, which is exactly the tradeoff the crawl was designed to avoid. The probe won't choose for you; it just prices it.
The same experiment in curl, one process, three transfers - note num_connects flipping to 0 after the first:
LM='Wed, 23 Sep 2026 02:32:16 GMT' # a real Last-Modified from this morning's fetch
curl -sS -o /dev/null -H "If-Modified-Since: $LM" \
-w "r1 code=%{http_code} t=%{time_total}s tls=%{time_appconnect}s conns=%{num_connects}\n" \
https://www.thordata.com/ \
--next -sS -o /dev/null -H "If-Modified-Since: $LM" \
-w "r2 code=%{http_code} t=%{time_total}s conns=%{num_connects}\n" \
https://www.thordata.com/
Results and the practical gotchas
Some things this exposed or confirmed while I ran it:
-
Send ETag when you have one; keep both when you can. Last-Modified is second-granular - edits inside the same second are invisible to it - and some origins quietly stop echoing
ETagon 304 responses, so refresh your stored validators only from full 200s. -
Weak validators are fine for caching, don't get clever.
W/"..."ETags are legal revalidation material (RFC 9111 lets you send them back); you only have to avoid using them for strong equivalence claims, like "these two fetches are byte-identical across variants". For dataset dedup, hash the body. - 304s spend the same rate-limit budget as 200s. If the target counts requests per IP - and most walls count requests, not bytes - your "polite" revalidation loop can exhaust a daily budget while fetching nothing. This is a server-design fact, not a measurement I made today; go verify your target.
-
304 may carry updated
Cache-Control. RFC 9111 says a 304's headers should update the stored response's headers. Most crawler caches ignore this. Yours doesn't have to. - The connection tax is also a gateway tax. Some per-session behaviors (sticky session TTLs, per-GB metering granularity, whether idle tunnels get reaped) determine whether "one warm connection per session" is even available as a strategy. Ask your provider what the tunnel idle timeout is; if it's shorter than your per-page thinking time, every revalidation silently paid a fresh handshake and your latency histogram has two peaks for a reason.
Where it lands: HTTP caching survives rotation in the byte dimension and breaks in the connection dimension. So price both. For high-value pages on per-GB lines, revalidate through a sticky session and let the low-value long tail rotate. For per-IP flat lines, the 304 is mostly theater if it costs you a fresh handshake each time - you could have spent that budget on the 200.
I write content for Thordata - both price points above were pulled from their pricing pages this morning, and the sticky-session mechanics referenced are what their residential gateway exposes (residential, datacenter); the probe code itself is vendor-neutral and worth running against whatever you use.
