Same API standard, 57 rivals, nobody consolidated: harvesting every Socrata portal at once

Quick answer

Several hundred US city, county and state governments publish open data through the Socrata Open Data API — the same /resource/<dataset-id>.json REST interface, the same SoQL query language, and a completely different set of columns per dataset. Most scrapers built on it hard-code one vertical. The Socrata Open Data Scraper is the generic case: you hand it a list of {portal_domain, dataset_id} pairs and it returns every matching row from every target in one run, whatever shape each dataset happens to be, at $2.00 per 1,000 results.

Why build a generic harvester instead of another vertical scraper? 📊

A quick market read, because it drove the design. Socrata has 57 competing listings on the Apify Store — but only 5 of them have real users, and the leader holds 5 users and 32 runs in 30 days. Wide and flat: demand is proven, and nobody has consolidated it. Every one of those 57 is a single-vertical wrapper — one city, one dataset, one fixed schema.

That is the gap. The protocol is uniform; the datasets are not. NYC's 311 complaints, Chicago's crime reports, Seattle's permits, LA's budget lines — same SODA interface, no two column sets alike. A product that only works if you already committed to one dataset is solving the easy half.

How do you return rows when you don't know the columns? 🧬

You stop pretending you do. The row model carries a small set of fields the harvester itself owns — which portal, which dataset, which target label produced this row — and passes the dataset's own columns through dynamically. No mapping table, no schema guessing, no silently dropping a column because it wasn't in a fixture.

The alternative — normalising everything into a fixed shape — is exactly what makes the 57 existing listings single-vertical. The moment you define a canonical column set, you have picked a dataset.

What does a Socrata portal do when the dataset isn't there? 🧱

It answers 404, and that is a routine outcome, not a crash. Dataset IDs get retired, portals reorganise, and a four-character typo in erm2-nwe9 is indistinguishable from a dataset that no longer exists.

If one bad target in a list of ten kills the whole run, you have built the single most common defect in this fleet: a recoverable per-item error taking down everything around it. So a 404 on one target is isolated — logged as dataset.missing, that target contributes zero rows, and the run continues to the next portal. Nine good targets still deliver nine targets' worth of data. That fault isolation is genuinely tested, not merely asserted in a docstring.

Verified on real portals, not on fixtures ✅

Both of these were hit live with curl before any code was written, and both are the shipped input prefill — so the demo run you get on first click actually returns rows:

A prefill that merely validates is not a demo. We have shipped an Actor whose every filter had a prefill and which ANDed them into a query matching nothing — it passed schema validation and failed cloud QA with "SUCCEEDED but produced 0 rows." A prefill has to return data.

The duplicate we caught before writing code 🔍

Our own fleet already contains an Actor with a Socrata client inside it — a building-permit lead-gen product. Same API family, different slug, and automated duplicate detection cannot see that: it diffs matching slugs and is blind to a same-API-different-slug clone. Three already-shipped targets got into our build queue that way once and were only caught at planning time by a human-style review.

So the distinction was written into the spec before implementation started, not assumed afterwards: the permit Actor is a fixed single-vertical product with a permit-shaped row for a lead-gen buyer. This one is a generic any-portal, any-dataset explorer with dynamic-schema passthrough for civic analytics. Different buyer, different output contract. If it had been the same product, the right outcome was to not build it.

FAQ

What does it cost?
$0.20 per run start plus $0.002 per row — about $2.20 for 1,000 rows. You pay for results, not runtime.

Do I need an API key or an app token?
No. Socrata's public endpoints answer keyless requests. App tokens only raise throughput limits; the Actor handles pacing and paging for you.

Can I filter server-side?
Yes. Each target accepts an optional soql_where clause (for example agency='NYPD'), so filtering happens at the portal rather than after you have already paid to download rows.

How many portals per run?
As many as you list. Page size and a per-target result cap are both inputs, so a run's cost stays predictable — and a target that 404s costs you that target, not the run.

Which portals work?
Any Socrata-hosted portal — several hundred US city, county and state governments, plus some EU ones. If it serves /resource/<id>.json, this reads it.

Built by Devil Scrapes. We do the dirty work so your dataset stays clean. 😈