Half of the scrapers we sell on Apify Store have a monitor switch: onlyNew: true. Schedule the actor, and every run returns only the rows that were not there last time. Class-action settlements that just opened, Kalshi markets that just listed, startups that just hit the marketplace. It is the one feature that turns a "try once" user into a daily schedule.
We shipped it to six actors in one afternoon. Unit tests green, cloud verification green: run 1 returned 136 rows, run 2 returned 0. Done.
Then we tried to write a regression test for it and realised our whole test harness could not express the feature.
The harness only knew about run 1
Every test we had followed the same shape: fresh storage directory, write INPUT.json, run src/main.js, read the dataset, assert. One run, one dataset. There was no way to say "now run it again against the same key-value store and check that the second dataset is empty."
That is not a small gap. The correctness of onlyNew lives entirely in the second run. Run 1 is indistinguishable from onlyNew: false. If the seen-set is never persisted, or is persisted under the wrong key, or is loaded but never applied, run 1 still looks perfect. The bug only exists in a state your harness never reaches.
So we added one field to the spec format:
{
"name": "onlyNew run-2 → 0 rows",
"input": { "onlyNew": true },
"runs": 2,
"minFirstRows": 50,
"minRows": 0,
"maxRows": 0
}
runs: 2 runs the actor twice in the same storage directory. Between runs the dataset is wiped but the key-value stores are kept — exactly what happens between two scheduled runs on the platform. minFirstRows guards the first run so a broken scraper that returns nothing twice does not pass as "correctly returned nothing new."
What the second run caught on day one
The new spec found no bug in the six actors. It found two bugs in how we thought about the feature, which is the more useful kind.
1. "0 new rows" depends on the cap. Our first draft of the spec inherited a base input with maxItems: 50 for the class-action actor. Run 1 returned 50 rows, run 2 returned 50 rows, and the check went red. Not a bug: the board has 194 settlements, the seen-filter runs before the cap (so unsent rows stay "new"), and run 2 correctly delivered the next 50. The golden case "second run is empty" is only true when the first run drained the board. If your cap is smaller than the source, the honest assertion is "run 2 is disjoint from run 1," not "run 2 is empty." We wrote the spec inputs so the board fits, and documented the disjoint case for later.
2. Some actors cannot be tested twice inside a sandbox call. The startup-marketplace scraper walks a 565-URL sitemap; one run is 50 seconds, two runs plus setup is over the time limit of the shell we get. We verified it by hand (409 → 0) and left a note in the spec. A test you cannot run automatically is a test that will silently stop being run; better to know that today than to discover it in a month.
The rule we took from it
If a feature's behaviour depends on state from a previous run — dedupe memory, trend baselines, "since last time" deltas — then the feature has no coverage until your harness can run the actor twice. Not "weak coverage." None. Run 1 exercises the code path that writes the memory, and that path is invisible from the dataset.
We now have seven runs: 2 specs across the catalog (kalshi, YouTube Shorts, MunchEye, class actions, the clipping-campaign aggregator, Microns, Skool). They cost a few seconds each and run with every maintenance pass.
If you sell a scraper with a monitor mode, go check when you last looked at its second run.
We maintain 45+ scrapers on Apify Store. Previous post in this series: Your {} smoke test passes. Your users still crash.
