Quick answer
Our new DirectIndustry Industrial Suppliers Scraper had a run that finished a third faster than the one before it, delivered the same 150 rows, and cost 41% more. Nothing failed. The only thing that changed was a memory setting, and the lesson is one line: on a per-GB-hour platform, memory is a rate, not a budget.
The Actor sells at $3.20 per 1,000 rows. The faster run cost $3.32 per 1,000 rows. It would have shipped at a loss before the developer share was even taken out.
What actually happened
DirectIndustry is a B2B catalog of industrial equipment: category pages fan out to product types, product types fan out to product pages, and each product page carries a manufacturer, a model and a specification table. The Actor walks that whole tree, which makes deep runs long. The first deep run took 3,006 seconds against a 3,600-second timeout. Too close.
The manifest had an invalid memory range (default 4,096 MB, maximum 2,048 MB), and the platform had been quietly clamping the run to 2,048. An earlier fix repaired the range by raising the maximum to 4,096, which also let the run take the memory it had been denied all along.
The next deep run finished in 2,011 seconds. Goal met. Then we read the bill.
| wall-clock | compute units | implied memory | cost per 1,000 rows | |
|---|---|---|---|---|
| before | 3,006 s | 1.67 | 2 GB | $2.35 |
| after | 2,011 s | 2.23 | 4 GB | $3.32 |
Cutting wall-clock by 33% while doubling memory raised cost by 41%. Compute is billed as memory multiplied by time. Doubling the first term and cutting the second by a third is a net increase, and the run had no use for the extra memory in the first place: the workload is curl-cffi fetching three product pages at a time. It is network-bound, and network-bound work does not get faster with RAM.
Why the first fix looked correct
Every signal we normally trust said the change was good. The run succeeded. The row count matched. The wall-clock improved. The TIMED-OUT risk was gone. A person reading the run log would approve it without hesitation.
The failure only exists in the ratio of two numbers on the usage record, and the usage record is not something a QA harness reads. So the second fix was boring: put the default back to 2,048 MB, leave the maximum at 4,096 so a customer can raise it themselves, and make a settled-usage cost measurement on a ~150-row run part of the pre-publish checklist, so the unshelve was gated on the ratio and not on the clock.
The check that gated the release
usage_usd = run["usageTotalUsd"] # read AFTER the run settles
rows = dataset["itemCount"]
cost_per_1k = usage_usd / rows * 1000
assert cost_per_1k < price_per_1k, f"sells at a loss: {cost_per_1k:.2f} vs {price_per_1k:.2f}"
Two details matter. The usage figure settles late, so an immediate read undercounts proxy transfer, which on this Actor is most of the bill. And a tiny smoke run is useless for this: the fixed per-run start charge dominates at 5 rows and the ratio looks terrible for every Actor. Measure at roughly a hundred rows, where the per-row cost is what is left.
What this means if you run scrapers on Apify
- Faster is not cheaper. Cost is GB-hours. Read the compute-unit count, not the duration.
- A network-bound crawler does not need more RAM. Give it more concurrency instead, and let wall-clock fall while the rate stays put.
- Repairing an invalid config can change behaviour. The platform was already clamping to the sane value. Making the manifest valid removed the clamp.
- Put the cost ratio in the gate. Price per 1,000 rows against measured cost per 1,000 rows, on a run big enough for the per-row term to dominate.
The Actor is live with the 2,048 MB default and the wall-clock budget guard that stops cleanly and flushes what it has before the platform kills it, so a deep category never becomes a TIMED-OUT run the customer still pays for. Rows carry the manufacturer, model, category, description, the full specification table and image URLs. Manufacturer website URLs are not included: DirectIndustry renders the supplier link without an outbound href on product pages, and we would rather say so than advertise a column that arrives empty.
