Building an x402-aware crawler in Go: what Cloudflare's Monetization Gateway means for web scraping

57% of internet traffic today is AI and web scraping bots — up from just 20% in 2021. For site owners, this has meant a frustrating binary: block bots using services like Cloudflare or Akamai and risk cutting off legitimate AI traffic, or allow them and get nothing in return.

x402 offers a third path. HTTP 402 — "Payment Required" — has existed in the spec since 1991 but was never practically implemented. x402 makes it real: a client hitting a payment-protected endpoint completes an on-chain transaction and attaches the payment signature (X-Payment-Signature) to the retry request before it can access the content.

x402 is an open, neutral standard for internet-native payments. It absolves the internet's original sin by natively making payments possible between clients and servers, creating win-win economies that empower agentic payments at scale.

It is part of the Linux Foundation, backed by Cloudflare, Coinbase, AWS, Google, Stripe, Visa, and others. This is not a crypto side project — it is foundational internet infrastructure.

Understanding the payment flow

The flow is straightforward:

The client requests a protected endpoint and gets back a 402 Payment Required response containing the price, destination wallet address, and expected network.
The client pays on-chain to that wallet via a facilitator, which handles the blockchain interaction.
The facilitator settles the transaction and returns a transaction hash.
The client retries the original request with the transaction hash as X-Payment-Signature.
The server verifies the payment on-chain through the facilitator.
On confirmation, the server serves the actual content.

On facilitators: a facilitator is an optional but recommended service that abstracts the verification and settlement process. You can use any facilitator as long as it supports the network the server expects payment on. Since Coinbase co-governs x402, most sites transact on Base, though Solana, Ethereum, and Polygon are also supported.

For testing x402-crawler, I used Base Sepolia (Base's testnet) and facilitator.x402.rs — a free public test facilitator run by FareSide, an independent open-source project. You can also run a local facilitator to avoid the blockchain entirely during development.

Architecture

Three concerns, deliberately kept apart: the client speaks the protocol, the crawler decides what to fetch, and the budget tracker decides what may be paid for. None of them holds another's state.

Package	Owns	Deliberately does not own
`client`	the x402 protocol: 402 handling, signing, retry classification, reconciliation	any spending policy, any ledger, any URL state
`crawler`	traversal: what to fetch, how deep, when to stop	how payments are signed
`crawler.BudgetTracker`	every spending decision and the accounting behind it	anything to do with HTTP

client.Config.Authorize is required — there is no default. A client with no policy would pay whatever a server asked, and that failure is silent and expensive. Callers who genuinely want no ceiling pass client.AllowAnyPayment, so the decision is written down rather than implied by an unset field.

Features

Simple BFS crawler, same-domain. Workers pull from a shared queue rather than traversing level by level, so one slow page does not hold up the rest. Termination counts queued and in-flight work — an empty queue is not the end, because a worker mid-fetch may be about to surface more URLs.

Budget-aware. A per-payment cap and a cumulative total, both claimed under one lock before signing, so concurrent workers cannot each be told there is room for the last payment. Running out of budget is not an error — the crawl reports which URLs it could not reach.

Pays when required, with retries. Every failure reason in the x402 spec is classified as terminal, retryable, or needs reconciling. A settlement_pending is never retried blind: the chain is asked what became of the transaction first, because retrying blindly is how one fetch gets paid for twice.

Dry run. --dry-run prices a crawl without funding it, still applying budget caps — so the estimate answers "what would this budget actually buy" rather than "what would everything cost".

PDF downloads. Streamed to disk rather than buffered, under a size cap, with filenames generated from a hash of the URL so a server cannot choose what lands on your filesystem.

Payment deduplication. Concurrent workers racing to fetch the same URL collapse onto one request via singleflight. The others wait and share the result rather than each paying independently.

Nothing leaks. A redacting zapcore.Core inspects log values — not field names — so a private key logged under any field name is still stripped from output.

The full implementation is on GitHub: github.com/HarishTeens/x402-crawler — give it a star if it's useful.

The bigger picture: Cloudflare's two-sided bet on crawler payments

Source: Cloudflare's article

I built for x402 — the open, wallet-native path. But Cloudflare is actually running two separate systems simultaneously, and understanding both clarifies why x402 matters.

The first is Pay Per Crawl. When Cloudflare announced the Monetization Gateway, most coverage missed that it sits alongside an older system with fundamentally different mechanics. Pay Per Crawl uses no blockchain. Cloudflare is the merchant of record: both the crawler operator and the publisher need Cloudflare accounts with billing set up. The crawler gets verified via Web Bot Auth, payments are settled through Cloudflare's billing infrastructure, and Cloudflare takes a cut. It is targeting enterprise — Perplexity, OpenAI, the players signing actual commercial deals.

The second is x402 via the Monetization Gateway. No Cloudflare account required. The wallet is the identity. Any agent can pay any site autonomously with no prior relationship. This is what I built support for.

These are not competing products — they are two halves of the same strategy. Pay Per Crawl locks in the enterprise market through billing relationships. x402 captures the long tail through open-protocol dominance. In both cases, Cloudflare sits in the middle of every transaction — as the billing layer in one, as the edge verification layer in the other.

That is the VISA parallel. VISA does not hold your money. It just sits between every transaction and charges a small fee. At Cloudflare's scale, with the majority of the web sitting behind their edge, that positioning is enormous — and it does not require owning the money to work.

As a crawler builder, x402 is the path to support now. It is autonomous, requires no prior relationship, and is the harder engineering problem — which is exactly what makes it worth building.