Four alarms fired overnight. Two database contention pages, one read-replica collapse, and a batch of metrics that came back empty.
Four alarms, one cause, and the cause was not anything we shipped.
The wrong first instinct
The reflex when four alarms land at once is to open the deploy log and start reading diffs. I did that for a few minutes. It is almost always wasted time, and there is a cheaper question that settles it.
Did request volume change?
workersInvocationsAdaptive, grouped by script, per hour
prod-api 3.6K/hr -> 18K/hr at 01:00Z
prod-web 3.6K/hr -> 17K/hr at 01:00Z
Both workers stepped up five times at the same minute and stayed there. A deploy does not do that. Traffic does. One query, and the entire code-review lane is closed.
The database side matched:
d1AnalyticsAdaptiveGroups, by databaseRole
primary reads 15-28K/hr -> sustained 37-46K/hr at 01:00Z
D1's primary is single-threaded. Roughly ten queries a second is about 36K an hour, so we were sitting above the ceiling. Everything downstream, the refusals, the replica page, the empty metrics, is one queue backing up.
The 50 seconds that named it
wrangler tail aiapplyd-prod-web --format json
Fifty seconds of live requests. 187 of them. What was in there:
- 155 of 187 from one hosting provider's VPS range, across eight or more addresses, geolocated to a single country.
- Plain desktop Chrome user agents, rotating across five major versions plus a macOS build. No bot user agent anywhere. Nothing in
robots.txtwould have stopped it, because it never asked. - 152 requests to
/jobs/{id}and 29 to/companies/{slug}. Nothing else.
That last line is the whole diagnosis.
Not one CSS file. Not one JS chunk. No font. No favicon.
A browser cannot render a page without them. A human cannot browse without a browser. So whatever this was, it was reading HTML and discarding it, at a rate of roughly four pages a second, walking two URL families in ID order.
The rotating user agent is what makes this worth writing down. It is a deliberate signal that the client wants to look human. The asset requests are the signal it forgot to fake, and they are free to check because they are already in your logs.
Why caching was not the answer
The first three suggestions in the room were all cache-shaped: longer TTL, more edge caching, a stale-while-revalidate window.
None of them help. The two families being walked hold hundreds of thousands of distinct IDs. Every request is a unique URL, on a first visit, at a rate the origin cannot absorb. A cache that never gets a second request is an extra hop. You cannot cache your way out of an enumeration.
What does work, in order of how quickly it can be deployed:
- Rate limit by ASN or by IP prefix, not by user agent. The user agent is attacker-controlled. The network path is not.
- Serve the enumerable families cheaply. If they are not indexable anyway, they do not need a database read per request.
- Verified-bot checks are not free. On our plan, the verified-bot field is enterprise-only. We reached for it, we could not have it, and knowing that in advance would have saved twenty minutes.
The two instrumentation lessons
Tail the worker that is actually being hit. Our API worker sees requests from our own frontend worker, which forwards without the original user agent. Tailing the API would have shown a wall of identical internal requests and nothing else. The client's fingerprint only exists at the true front door. Name the process that receives the traffic, then tail that one.
And check the alarm's window before you believe it. One of these alarms re-fired the next morning at full severity for an incident that had been fixed the previous afternoon, because it computed its rate over a rolling 24 hours. The burst was inside the window. The verdict was stale.
A rate wants a wide window. A verdict wants a recency gate. If you only have one window, you get paged for your own fixes, and the day you stop trusting the page is the day it is real.
I build AI Applyd. We scrape job listings ourselves, on a strict budget and only from sources that allow it, which is exactly why I recognise the shape of a client that does not read your CSS.
