Quick answer
We swept 288 of our own Apify Actors for a specific, boring defect: a column that is empty in every single row. Thirty-five had at least one. Two were serious enough to pull a developer onto the same morning.
The sweep costs nothing — it reads the dataset each Actor already produced on its last successful run and starts no new runs. It found a live, paid property scraper returning listings with no bedroom count, no coordinates and no images, where every run had been reporting SUCCEEDED for weeks.
Why doesn't a normal test catch this?
Because nothing about it looks like a failure. The run exits zero. The row count is healthy. The schema validates, because null is a permitted value for an optional field. Every gate you have — exit status, row count, schema check, smoke test — is answering a different question than the one that matters, which is did the customer get the data we advertised?
The specific trap is that an optional field and a broken field are indistinguishable from one row. bathrooms: null is completely normal for a studio flat. bathrooms: null on all twenty rows means the selector is dead. The signal only exists in aggregate, and nobody looks at datasets in aggregate — they look at one row, confirm it "looks right", and move on.
So the check is trivial once you state it: for every field, is it empty in every row?
fields = set().union(*(set(r) for r in rows))
dead = [f for f in fields
if all(_is_blank(r.get(f)) for r in rows)]
The hard part isn't the code. It's deciding what the hits mean.
How do you tell a real defect from a sampling artifact?
Weight by sample size. That single rule did most of the work.
A dead column across 5 rows is noise. An eBay scraper showing bids_count: null across five rows sampled from fixed-price listings is correct behaviour. A Trustpilot scraper with company_reply_text: null across six reviews nobody replied to is correct behaviour. Chasing those wastes a developer and — worse — trains everyone to ignore the alert.
A dead column across 90 or 150 rows is a defect. Our two real hits:
-
A property scraper: 17 dead fields across 20 rows. Rooms, bathrooms, floor, latitude, longitude, agency name, images, description — all null, and
location_citywas the literal string"Unknown". - An ad-library scraper: the advertiser name null across 90 rows. On a product whose entire purpose is "which ads is this brand running".
Both had shipped. Both were being paid for.
What were the actual root causes?
Neither was a regression. Both were never implemented, and that turned out to be the more interesting finding.
The property scraper's parser had been written against a fixture that invented a wire format the site does not serve. A live capture found zero application/ld+json and zero of the #agency-data blocks the parser was reading. The fixture described a fiction, the parser matched the fixture, and the unit tests passed forever because fixture and parser shared the same wrong model. Tests confirm your code does what you think — they cannot tell you that what you think is wrong about the outside world. Only a real response can.
The ad-library scraper's advertiser_name was a pure echo of the customer's own search filter. If you searched by advertiser, it came back populated — with the string you typed. Leave the filter blank and the column was null by construction, permanently. The API genuinely does not return advertiser identity to anonymous callers; we confirmed that across 227 rows and nine countries before concluding it. The real identity was one level down, embedded in each ad's public detail page.
The sting in the tail: fixing it isn't enough
Both fixes landed, and both Actors still reported the dead column on the next cloud test.
The reason is worth knowing if you build on Apify or any similar platform. The recovered fields sat behind an opt-in flag — reasonable, because they cost an extra fetch per item. But the platform runs its automated check against the Actor's prefill, and a customer clicking Try runs the prefill too. The flag was off there. So the fix existed, the tests passed, and the run a prospective buyer would actually see still returned the same hollow row.
A feature that is off in your prefill does not exist as far as your automated checks or your customers are concerned. We enabled it in the prefill — bounded to a handful of results so the demo stays cheap — and kept the default off so an API caller who omits the flag doesn't silently buy the extra cost.
The takeaway
An advertised column that is permanently empty is worse than an absent one. It passes every check, it survives code review, and it is discovered by the person paying for it.
So: either populate it, or stop advertising it. We did both, depending on the field — six of the property scraper's columns genuinely aren't available from the source on any of its regional sites, and those are now documented as null rather than implied to work.
Sweep your own outputs in aggregate. It costs nothing and it is the only way this class of bug is visible.
Built by Devil Scrapes. The two Actors above are the Idealista Property Scraper and the Pinterest Ad Library Scraper — both fixed, both with the recovered fields now exercised by their default run.
