Cloudflare Is Silently Blocking AI Crawlers on Your Webflow Site. Here Is How to Fix It.

Why Is Your Webflow Site Invisible to ChatGPT and Perplexity?

Every Webflow site now runs on Cloudflare infrastructure following the mandatory hosting migration completed in April 2026. Cloudflare provides faster global performance, superior DDoS protection, and improved security. But it also comes with a default setting that may be silently blocking the very AI crawlers you need visiting your site. Cloudflare's Bot Fight Mode, which is enabled by default on all plans including the Free tier, blocks automated traffic that it classifies as bots. That classification includes legitimate AI crawlers like GPTBot (OpenAI/ChatGPT), PerplexityBot, ClaudeBot (Anthropic), and Google-Extended (Google Gemini).

This means your Webflow site could be ranking well in traditional Google search, producing excellent content, and implementing every AEO best practice, yet remain completely invisible to AI-powered answer engines because the crawlers never reach your pages. They hit the Cloudflare layer, get classified as bots, and receive a block or challenge response before your server ever sees the request. Your access logs show nothing because the traffic never arrives.

I discovered this issue on a client site in early 2026 when Perplexity citations for pages that clearly should have been cited were consistently missing. The content was strong. The schema markup was correct. The robots.txt was clean. The problem was Cloudflare, and it took weeks to diagnose because the blocking happens silently at the network level. Here is how to check and fix it.

How Does Cloudflare Bot Fight Mode Work?

Cloudflare Bot Fight Mode is a security feature designed to protect websites from malicious automated traffic, including credential stuffing attacks, content scraping bots, spam bots, and DDoS attempts. When enabled, Cloudflare analyzes incoming requests using machine learning, behavioral analysis, and fingerprinting to determine whether the visitor is a human using a browser or an automated bot. Requests classified as bot traffic receive either a block response, a JavaScript challenge, or a managed challenge that prevents the bot from accessing the page.

The problem is that Bot Fight Mode does not distinguish between malicious bots and legitimate AI crawlers. GPTBot, PerplexityBot, ClaudeBot, and other AI service bots are automated systems that send HTTP requests, which is exactly the behavior Bot Fight Mode is designed to detect and block. Cloudflare recently changed its default configuration to be more aggressive against AI bots specifically, meaning sites that previously allowed AI crawlers may now be blocking them without the site owner's knowledge.

For Webflow sites, this is particularly problematic because the Cloudflare configuration is not directly accessible through the Webflow dashboard. Webflow manages the Cloudflare integration at the infrastructure level. However, sites using Cloudflare Orange-to-Orange (O2O) configuration, where the site owner also has their own Cloudflare account layered on top of Webflow's Cloudflare, have direct access to Bot Fight Mode settings through their Cloudflare dashboard.

How Do You Check If AI Crawlers Are Being Blocked?

The most reliable method is checking your server logs or Cloudflare analytics for AI bot user-agent strings. If you have access to a Cloudflare dashboard (through O2O setup), navigate to Security, then Bot Analytics. Look for requests from user agents containing GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Anthropic-AI, and Google-Extended. If you see these user agents with "blocked" or "challenged" status, Bot Fight Mode is preventing AI crawlers from reaching your content.

Cloudflare also offers an "AI Crawl Metrics" page in the dashboard that specifically shows AI bot traffic to your site. This page breaks down requests by AI crawler, showing how many requests each bot made and whether they were allowed, challenged, or blocked. If this page shows zero requests from major AI crawlers, either the crawlers are not visiting (possible) or they are being blocked before the requests register (more likely if Bot Fight Mode is enabled).

If you do not have direct Cloudflare dashboard access (which is the case for most standard Webflow hosting setups), an indirect test is to search for your site's content on Perplexity or ChatGPT. Ask a specific question that your content definitively answers. If the AI platform does not cite your page despite it being the most authoritative answer on the web for that query, the crawlers may not be able to access your content. This is not a definitive test since AI citation depends on many factors, but it is a useful signal when combined with other checks.

Another approach is to check the ChatGPT-User bot specifically in your Webflow Analyze traffic sources. If Webflow Analyze shows zero traffic from AI platforms despite having content that should be cited, the Cloudflare layer may be intercepting the crawlers before they reach Webflow's tracking.

How Do You Allow AI Crawlers Through Cloudflare?

The fix depends on your Cloudflare setup. If you have your own Cloudflare account with Orange-to-Orange (O2O) configuration, the most direct solution is to create firewall rules that explicitly allow known AI crawler user agents. In your Cloudflare dashboard, go to Security, then WAF, then Custom Rules. Create a new rule that matches requests where the user agent contains GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Anthropic-AI, or Google-Extended, and set the action to "Allow" or "Skip" for Bot Fight Mode.

You can also disable Bot Fight Mode entirely from Security, then Bots, then Bot Fight Mode toggle. However, this removes protection against all automated traffic, not just AI crawlers. The firewall rule approach is more surgical because it allows specific AI bots while keeping Bot Fight Mode active for everything else.

If you are on standard Webflow hosting without a separate Cloudflare account, your options are more limited. Webflow manages the Cloudflare configuration at the platform level, and individual site owners cannot directly modify Bot Fight Mode settings. In this case, your primary control is through the robots.txt file. Ensure your robots.txt explicitly allows AI crawlers by not including any Disallow rules for GPTBot, PerplexityBot, ClaudeBot, or Google-Extended. While this does not override Cloudflare's network-level blocking, it ensures that any AI crawlers that do reach your server are not blocked at the application level.

For sites where AI visibility is business-critical, setting up Cloudflare O2O specifically to gain control over bot management settings is worth the effort. The O2O setup is free on Cloudflare's Free plan and gives you direct access to the security controls that determine whether AI crawlers can reach your content.

Which AI Crawlers Should You Allow and Which Should You Block?

Not all AI bots serve the same purpose. Some crawl your site to generate real-time answers for users (which drives traffic back to you through citations). Others crawl your site to train their models (which uses your content without sending traffic). Understanding the difference helps you make informed decisions about which bots to allow.

GPTBot is OpenAI's crawler used for both training and real-time retrieval. ChatGPT-User is the bot that fetches pages in real time when a ChatGPT user asks a question that requires current information. Allowing ChatGPT-User is essential for AI citation because it is the bot that generates the citations that send traffic to your site. PerplexityBot crawls pages to provide real-time answers with source links. ClaudeBot is Anthropic's crawler. Google-Extended is Google's AI training crawler, separate from Googlebot which handles traditional search indexing.

The recommended approach for most business websites is to allow all major AI crawlers. The traffic these bots generate through citations (which Webflow's data shows converts 6x better than organic) far outweighs any concern about content being used for training. If you specifically want to allow real-time retrieval bots while blocking training crawlers, you can create more granular firewall rules, but the practical distinction is difficult to enforce and the trade-offs rarely justify the complexity for small business sites.

What About robots.txt for AI Crawlers on Webflow?

robots.txt is your application-level control for crawler access. Webflow auto-generates a robots.txt file for your site, but you can customize it through Site Settings. Check your current robots.txt by visiting yourdomain.com/robots.txt in your browser. If you see any rules that block AI crawlers (User-agent: GPTBot followed by Disallow: /, for example), those rules are preventing AI bots from accessing your content even if Cloudflare allows them through.

For maximum AI visibility, your robots.txt should not contain any Disallow rules for GPTBot, PerplexityBot, ClaudeBot, ChatGPT-User, Anthropic-AI, or Google-Extended. If you want to allow all AI crawlers, the simplest approach is to ensure there are no blanket Disallow rules and that the default Webflow robots.txt configuration is unchanged.

Keep in mind that robots.txt and Cloudflare operate at different layers. Cloudflare acts at the network level and can block a request before it ever reaches your server. robots.txt acts at the application level and only affects crawlers that successfully reach your server. For AI visibility, you need both layers configured correctly. If Cloudflare blocks the crawler, your permissive robots.txt is irrelevant because the crawler never sees it.

How Does This Connect to Your AI Visibility Strategy?

Allowing AI crawlers to access your site is the absolute foundation of any AEO strategy. You can have perfect schema markup, comprehensive answer blocks, excellent internal linking, and authoritative content, but if the AI crawlers cannot reach your pages, none of it matters. Crawler access is step zero.

The research is clear on the value of AI-referred traffic. Webflow reports that AI traffic converts 6x better than unbranded organic search. Ahrefs found that AI search visitors convert 23x higher than traditional organic visitors on their platform. Blocking AI crawlers, whether intentionally or through default Cloudflare settings, means shutting off access to a traffic channel that converts dramatically better than traditional search.

Once crawler access is confirmed, the rest of your AI visibility stack becomes effective. Your llms.txt file can guide AI crawlers to your best content. Your schema markup can provide structured data for AI extraction. Your answer blocks can be surfaced in AI-generated responses. Your internal linking architecture can help AI systems navigate deeper into your content across multiple conversation turns. But all of it requires that the crawlers can actually reach your pages in the first place.

What Should You Do This Week?

Check your robots.txt by visiting yourdomain.com/robots.txt. Confirm there are no Disallow rules blocking AI crawler user agents. If you have a Cloudflare account through O2O setup, log in and check your Bot Fight Mode settings and AI Crawl Metrics. Create firewall rules to allow known AI crawlers if they are being blocked.

If you do not have a separate Cloudflare account, test your AI visibility by searching for your content on Perplexity and ChatGPT. If your content should be cited but is not, consider setting up Cloudflare O2O to gain direct control over bot management settings. The setup is free and takes about 30 minutes.

For the full AEO strategy that depends on crawler access, my tutorial on getting your Webflow site cited by ChatGPT, Perplexity, and Google AI covers every step. For the llms.txt file that guides AI crawlers once they can access your site, my guide on setting up llms.txt on your Webflow site walks through the implementation. And for understanding why AI traffic is worth fighting for, my article on why AI traffic converts 6x better than organic covers the conversion data.

AI crawlers are the new search engine spiders. Blocking them is like blocking Googlebot in 2010. If your Webflow site is invisible to ChatGPT, Perplexity, and Claude, you are missing the highest-converting traffic channel available in 2026. The fix takes 30 minutes. If you want help auditing your site's AI crawler access or configuring Cloudflare for AI visibility, I am happy to walk through it. Let's chat.