What Does Cloudflare's Shift From Pay Per Crawl to Pay Per Use Mean for Webflow Owners?

What did Cloudflare just change about AI crawlers?

Cloudflare is moving from charging AI crawlers each time they fetch a page to charging when the content actually creates value. It calls the new model pay per use, and it launched a Monetization Gateway to collect those payments. The shift builds on its earlier pay per crawl idea, which it now says did not go far enough.

This matters to anyone who publishes on the open web, and that includes site owners on Webflow. Cloudflare sits in front of a huge slice of the internet, so its rules on AI traffic shape how bots reach your pages. When Cloudflare changes the deal, the whole content economy feels it.

I have been watching this closely because it touches both sides of my work, AI search and web infrastructure. Let me break down what changed, using Cloudflare's own announcement, and what it means for you.

What is the difference between pay per crawl and pay per use?

Pay per crawl charged an AI company each time its bot fetched a page. Pay per use charges based on value created, not just on the fetch. Cloudflare now says a bot grabbing a page is the wrong moment to bill, and that the better moment is when your content shows up inside an AI answer.

The logic is fair. A crawler might fetch your page once and reuse it in thousands of answers. Under the old model, you got paid for the one fetch. Under the new one, Cloudflare frames payment around the ongoing value the content delivers, per its blog. That aligns the fee with the real benefit.

This is a real change in thinking, not a rename. It reframes your content as a resource that keeps earning, rather than a one time download. I wrote about the original model in my post on what pay per crawl meant for Webflow owners, and this is the next step in that story.

What is the Monetization Gateway?

The Monetization Gateway is Cloudflare's new system that lets publishers charge any caller for almost any resource, without building the payment plumbing themselves. Per Cloudflare, that covers a web page, an API, a dataset, or even a tool call from an AI agent. It handles the billing so you do not have to.

This is the part I find most interesting. It turns Cloudflare's edge into a tollbooth you can switch on. Instead of only blocking or allowing bots, you can ask them to pay. For publishers who felt powerless as AI scraped their work for free, that is a real lever.

It also fits a wider trend. As AI agents start doing tasks on the web, more of the traffic to your site may come from software, not people. A gateway that can charge those agents gives content owners a way to get value from a visitor who never sees an ad or fills a form.

How does x402 and the 402 status code fit in?

The payments run on an open protocol called x402, named after the HTTP 402 status code. That code, labeled Payment Required, has sat mostly unused for years. Cloudflare is finally putting it to work as the signal that a resource costs money to access.

Here is how Cloudflare describes the flow. When a client asks for a protected resource, the edge can return a 402 response with a machine readable price. Per Cloudflare, that manifest lists the cost in the USDC stablecoin, the accepted blockchain, which is Base or Solana at launch, a destination wallet, and a payment timeout.

The choice of an open standard matters more than the crypto detail. If the way to charge machines becomes a shared protocol, other providers and agents can speak it too. That is how a one company feature can grow into a web wide habit, which is what makes this news bigger than a single product.

What changes on September 15, 2026?

Starting September 15, 2026, Cloudflare changes its default settings for new domains. Per its announcement, the Training and Agent crawler categories will be blocked by default on pages that display ads, while the Search category will stay allowed by default. So the defaults now lean toward protecting monetized content.

To make that work, Cloudflare now sorts AI crawlers into three buckets, Search, Agent, and Training. Search bots that send visitors your way stay welcome. Training bots that feed models and Agent bots that act on your content face a stricter default on ad pages. You can block or charge each type on its own.

This is a meaningful default flip. For years the default was open, and blocking took effort. Now, for ad supported pages on new domains, the burden shifts toward the crawler. Cloudflare says its customers already send more than one billion of these 402 responses to AI crawlers every day, which shows the scale in play.

Does this affect my Webflow site directly?

It depends on how your site sits behind Cloudflare. Many Webflow sites use Cloudflare in front of their custom domain for speed and security, and those are the ones most directly touched. If you run that setup, the new crawler categories and controls become tools you can actually reach.

If you use Webflow hosting without your own Cloudflare account in front, the change is more of a signal than a lever for now. You cannot flip Cloudflare's new switches directly, but the industry shift still shapes how AI companies treat your content and how the market prices access to it over time.

Either way, the smart move is to know how AI bots reach your pages today. That means understanding your crawler access rules, which I cover in my guides to verifying AI crawlers on Webflow and to setting up an llms.txt file for AI crawlers.

Should I try to charge AI crawlers now?

For most small business sites, not yet. The Monetization Gateway is powerful, but it fits publishers whose content has clear standalone value at scale, like large media libraries or datasets. If your site is a lead generating brochure or a service page, blocking or shaping bad bots matters more than billing them.

My honest take is that the bigger win for founders is visibility, not tolls. You usually want the Search category to keep reaching you, since that is what puts you in front of buyers. Charging every bot could quietly cut you out of answers you actually want to appear in. That trade is rarely worth it for a small brand.

So I would not rush to switch on tolls. I would watch the categories, keep search access open, and decide about training and agent bots case by case. The point is to make an active choice instead of leaving it to a default you never reviewed.

What does this mean for AI search visibility?

It means the open web is starting to put a price on AI access, and that will slowly reshape which sites get seen and cited. If more publishers block or charge training and agent bots, the pool of freely readable content shrinks. Sites that stay open and well structured may get relatively more attention from answer engines.

That is a real opportunity for founders who want to be found. While big publishers wall off content, a fast, open, well marked up Webflow site can remain easy to crawl and quote. Being generous with access, on your own terms, can become a visibility edge in an era of rising walls.

It also raises the value of getting the basics right. Clean structure, clear answers, and correct crawler rules decide whether AI engines can use your content at all. The pricing debate is loud, but the quiet work of being readable still wins most of the citations.

What should you do next?

Start by finding out whether Cloudflare sits in front of your site and how your AI crawler rules are set today. Decide, on purpose, which bots you welcome and which you limit. Keep search access open if you want AI visibility, and treat charging as a later option, not a first move.

This is a fast moving space, and the defaults are shifting under everyone. The owners who come out ahead will be the ones who understand the rules early and make deliberate choices, rather than reacting after a policy already changed their traffic. A little attention now saves a scramble later.

If you want help sorting out how AI bots reach your Webflow site and what to do about it, reach out. I help founders keep their sites visible to the right crawlers while blocking the ones that only take. Let's connect and get your setup ready for this shift.