What happened to TheNumbers.com

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site.

Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.

At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come.

Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.

What does a cyber attack have to do with AI scraping?

I think people are missing one of the points of the article. It's not just that agents are hammering the site, it's that there might be lurking vulnerabilities that allow malicious usage, which is why it went down, then came back up with a fraction of the data and a reduced design.

The article says (speculates?) that malicious users are trying to get privileged access for an edge in prediction market betting. From the article:

> If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders - learning the answers slightly ahead of publication would allow you to front-run the trades.

"One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products."

I know this isn't really the point of the article, but I've been thinking about this a lot. I wonder if we're going to see more resources go this way.

I used to publish little doo-dads as opensource software. Not because it was something that was legitimately ground breaking or anything (it absolutely wasn't) but because I had a problem, and I thought "heh wouldn't it be cool if someone else had a similar problem and could use my resource for it."

But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way.

And I can see why maintainers of sites like this, or other free but incredibly useful resources might start to get irate at that.

What a throwback. Back in 2015 I have started Applaudience, which at the time was the only provider of real-time cinema ticket sales data. I still remember comparing our numbers against TheNumbers.com as part of calibration. I have since moved on to other businesses, but this remains one of my favorite pieces of technology that I have developed. Would love to bring it back one day.

I know people have strong opinions about Cloudflare but why not use it here, at least as a stop gap? Stopping bot traffic is one thing it does very well.

If the AI companies destroy the open web, eventually they'll need to start curating knowledge sources just like netflix makes movies and amazon has physical stores...