We need better than Pay-to-Crawl

The data-hunger of current AIs is reviving an interesting old idea: pay-to-use internet (pay-to-crawl in this case). This development could have very positive ramifications in principle, but current proposals worry me.

What is happening? Both the training and operation of popular AI models use vast quantity of data scraped from the internet. This comes at a cost to the provider of said data, who needs to pay for the infrastructure and the electricity needed to serve it. This cost is worthwhile if the data goes to a human, but not if the recipient is an AI. Most internet traffic is now comprised of bots [10].

Obviously, website owners often want to block or limit these bots [2,3]. The problem is that some bots have taken shady measures to pass as human. They ignore the robots.txt file, they change their user-agent, and most worryingly, they run on botnets of consumer PCs without those users' knowledge [1]. Install the wrong browser plugin and your PC could be scraping data for AI training at your expense and without your knowledge.

With bots so difficult to keep out entirely, a few companies and non-profits are instead tying to make them pay to access their websites (pay-to-crawl) [5,6,7,8,9].

As a concept, this is a good idea. Paying for access was how the internet was originally envisioned. No suitable payment system existed yet, and eventually ads filled the niche, but they came with several toxic incentives. Multiple people, including myself, have created systems to replace ads with micropayments, but none gained traction. The network effect of ad-based platforms is too strong. There is no network effect yet for AI bots, so maybe pay-to-crawl will be the incentive that finally enables pay-to-use internet. If that becomes the case, great! It means we are at the dawn of a fairer and more productive internet.

Current proposals do not fill me with confidence.

They are voluntary, automated licensing frameworks, not a way to prevent bots from pretending to be humans. The license they provide may not even be legally required for AI training, as US courts have repeatedly ruled that training falls under fair use. As such, from a technical standpoint, pay-to-crawl's primary use case seems to be AI-exclusive data libraries. Like a Wikipedia that is only accessible through Gemini. Ouch.

While pay-to-crawl cannot distinguish between humans and bots, there is a technological arms race to do just that. Cloudflare is the largest company providing a bot-exclusion service and it is also experimenting with a pay-to-crawl system. They could force at least some bots to pay the crawling fee. I consider this solution to be even worse, as it gives too much power to one company. Already, Cloudflare controls 20% of the internet.

The better solution is to require a micropayment covering the hosting cost from every user, bot or human. This cost is so low that it does not matter to humans, as we who consume data relatively slowly. The systems we currently use to keep the bots out are not free either; they cost time and sometimes compute/electricity.

Creative Commons has expressed worry about pay-to-crawl [4], but their stance is a fantasy. They wish for content to remain free to humans. If it is free to humans, it is free for bots pretending to be humans.

References: