Cloudflare Will Block Mixed-Use AI Crawlers by Default on September 15: What It Means for Your Site

What did Cloudflare just change about AI crawlers?

On July 1, 2026, Cloudflare announced that starting September 15, 2026, its default settings will block "mixed-use" AI crawlers from any page that hosts ads. It also announced that Pay Per Crawl is evolving into Pay Per Use. In short, Cloudflare is making it easier for site owners to control, and get paid for, AI access to their content.

Cloudflare sits in front of a huge share of the web, so when it changes a default, a lot of sites change with it. This announcement, made on what Cloudflare calls Content Independence Day, is one of the bigger moves yet in the fight over who controls AI access to the open web.

I have been watching this shift closely because it touches everyone I work with. Here is what actually changed, who it affects, and what I think you should do about it.

What exactly did Cloudflare announce on July 1, 2026?

Cloudflare announced two things. First, from September 15, 2026, its default settings will block mixed-use AI crawlers on pages that host ads. Second, it is turning Pay Per Crawl into Pay Per Use, so publishers get paid when their content creates value, not just when a bot fetches it. Both were shared on Cloudflare's own blog and press release.

The news was widely reported by outlets like TechCrunch and Forbes on July 1, 2026, and it lines up with Cloudflare's stated philosophy of "your content, your rules." The company has spent the last year building tools that let site owners decide how AI systems may use their pages, and this is the next step.

According to Cloudflare's announcement, the default block applies to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. That scope is what makes it matter. A default that reaches free customers touches a very large number of small sites at once.

What is a mixed-use AI crawler, and why does it matter?

A mixed-use crawler is a bot that gathers content for more than one purpose at once, such as both search indexing and AI training. It matters because you may want a bot for search but not for AI training, and a mixed-use bot forces an all-or-nothing choice. Cloudflare is pushing AI companies to separate those jobs.

For years, the deal was simple. You let Googlebot crawl your site so you could show up in search. Now dedicated AI crawlers like OpenAI's GPTBot and Anthropic's ClaudeBot, along with mixed-use bots, might feed an AI model that answers questions without ever sending you a visitor. When one bot does both search and AI, blocking the AI use also risks your search visibility.

Cloudflare's move gives AI companies until September 15, 2026, to split those crawlers apart, or be blocked by default on many publisher pages. The goal is to let site owners say yes to search and no to AI training, which is a distinction site owners have wanted for a long time. I dug into the tradeoffs in my note on setting a robots.txt strategy for AI bots.

Who does the September 15 default block actually affect?

The default block mainly affects pages that host ads, and it applies to new Cloudflare customers, new sites, and all existing free customers. If your site runs on Cloudflare, is ad-supported, and you have not changed the defaults, this change can start blocking mixed-use AI crawlers for you automatically on September 15, 2026.

That word "default" is the key. Most people never touch their crawler settings, so a default is what actually shapes the web. By flipping the default for free customers and new sites, Cloudflare changes behavior for millions of pages without anyone lifting a finger.

Publishers who sell ads are the clearest winners here, because their content is being used to train models that then compete with them for attention. But even if you do not run ads, this is worth understanding, because defaults like this tend to spread and set expectations across the whole web.

What is Pay Per Use, and how is it different from Pay Per Crawl?

Pay Per Crawl charged AI companies each time they fetched your content. Pay Per Use pays publishers when their content actually creates value, such as when it appears in an AI answer. It is a shift from charging for access to sharing in the outcome, which better matches how AI systems really use content.

Cloudflare launched the first version of Pay Per Crawl about a year earlier, on the first Content Independence Day. The idea was straightforward: if a bot wants your content, it can pay to crawl it. Useful, but crude, because a fetch is not the same as value created.

Pay Per Use tries to fix that. Cloudflare said it is starting with two partners, Ceramic.ai and You.com, so a publisher can be paid when their content shows up in those AI results or is accessed as premium content. It is early and small, but the direction is clear: content owners want a cut when AI answers are built on their work. This continues the story I told in my piece on what Pay Per Crawl means for site owners.

Does this affect my Webflow site if I do not run ads?

If you do not run ads, the September 15 default block is unlikely to change anything for you directly, especially since it targets ad-hosting pages. But the broader shift still matters, because it signals that controlling AI access is becoming a normal, expected part of running any website.

Most of the founders and businesses I work with do not sell display ads. Their sites are portfolios, services pages, and blogs meant to attract clients. For them, being crawled by AI is often good, because it can lead to being cited and recommended in AI answers.

So I would not rush to block AI crawlers on a lead-generating site. The real lesson here is awareness and choice. You should know which bots reach your site, decide on purpose whether you want AI access, and set your rules deliberately rather than by accident. I walked through the practical side in my guide on blocking AI crawlers on a Webflow site the right way.

Is this good or bad for small site owners?

On balance, I think this is good for small site owners, because it gives them control and a possible path to payment that they did not have before. The risk is complexity and confusion, since defaults that block crawlers can also hurt discovery if you are not paying attention. Control is only good when you understand it.

The upside is real. For years, AI companies took content freely, and creators had little say. Tools like this rebalance that, letting owners set terms and even earn from AI use. That is a healthy correction, and I welcome it.

The risk is that people block too much without thinking. If you depend on being found and cited, blocking every AI bot by reflex can quietly cut you off from a growing source of visibility. The right move is deliberate choice, not fear. Know what each setting does before you flip it.

What should I do before September 15?

Check whether your site runs on Cloudflare, whether you host ads, and what your current AI crawler settings are. Decide on purpose whether you want AI systems to access your content. Then set your rules to match that choice, rather than letting a default decide for you. A short audit now prevents surprises later.

Start by confirming your setup. If you are on Cloudflare, log in and look at your bot and AI crawler controls so you know your current state. If you are not on Cloudflare, this specific change does not apply, but the same questions still do.

Then make an intentional decision. If you want AI visibility, keep useful crawlers allowed and make your content easy to cite. If you want to protect and monetize your content, explore the new controls and Pay Per Use. Either way, the goal is a choice you made on purpose, aligned with how you actually earn from your site.

What does this signal about the future of AI and the open web?

It signals that the free-for-all era of AI crawling is ending, and that content owners are gaining real tools to set terms. The web is moving toward permission and payment for AI use, not just open access. Whether you love or hate that, it is becoming the new normal, and planning for it beats ignoring it.

For most of the web's history, crawling was a fair trade: bots took your content, and search sent you visitors back. AI broke that trade, because it can answer from your content without sending anyone your way. Moves like Cloudflare's are the web trying to rebuild a fair exchange for the AI era.

I expect more of this, not less. Other platforms and hosts will add similar controls, and deciding how AI may use your content will become a routine setup step, like choosing a privacy policy. The businesses that think about this early will have more control and more options than the ones who wait.

What should you do next?

Do not panic, but do pay attention. Check your Cloudflare and crawler settings, decide whether AI access helps or hurts your goals, and set your rules on purpose before September 15, 2026. For most lead-focused sites, staying visible to AI is still the right call, but the choice should be yours.

This is a fast-moving space, and the right answer depends on how your specific site earns its keep. If you want help deciding whether to welcome or limit AI crawlers on your site, and how to stay visible in AI answers either way, reach out through pravinkumar.co and let's talk it through.