Is Your CDN Migration Quietly Blocking Googlebot?

Is your CDN migration quietly blocking Googlebot?

It might be. In November 2026, Google Search Central published guidance warning that a new CDN can trigger a crawl surge against your origin, and that blanket bot protection can block verified Google crawlers. If crawlers keep hitting challenge pages or errors, Google says pages may drop out of Search.

This pattern is most common on B2B marketing sites that move behind Cloudflare, Fastly, or Akamai for speed or security, usually right before a launch. The person flipping the switch is thinking about performance and DDoS protection. Nobody on the call is thinking about Googlebot.

The post was written by Gary Illyes and Cherry Prommawin, and it accompanies an episode of Google's video series The Non-Canonical Cabinet. It is short, practical, and worth ten minutes of any revenue team's time, because organic and AI search traffic feed pipeline long after the migration ticket is closed.

What did Google actually say about CDNs and crawling?

Google made three points. Faster, more stable responses raise the crawl capacity limit, so crawlers request more pages after a CDN move. A new CDN starts with a cold cache, so early crawls fall through to your origin. And broad bot blocking or CAPTCHA challenges can stop verified Google crawlers.

The first point surprises people. You move to a CDN to make the site faster, and Google responds by crawling harder. The post describes this through the crawl capacity limit, which Google also calls hostload. When your server answers quickly and reliably, Google reads that as permission to ask for more.

The second point explains why that extra crawling hurts. The post puts it plainly: "Caches only shield you on the second visit." Human visitors cluster on a small set of popular pages, so those warm up fast. Crawlers walk your whole URL inventory, including old blog posts, tag pages, and CMS items nobody has visited in a year. Every one of those first requests is a cache miss that lands on your origin.

The third point is where real damage happens. If your firewall or CDN challenges every unfamiliar bot, Googlebot gets the challenge page instead of your content.

Why does this matter for a B2B go-to-market team?

It matters because organic search and AI answers are pipeline channels, not vanity metrics. If Googlebot sees challenge pages for days after a migration, your comparison pages, pricing page, and case studies can fall out of results that buyers use to build a shortlist, and you will not see it in your CRM until later.

Here is the part that worries me as a GTM engineer. The damage shows up with a delay. A launch goes live, the site feels fast, and the team celebrates. Two weeks later, inbound demo requests from organic dip, and nobody connects it to a firewall toggle that someone enabled during the cutover.

I treat infrastructure changes on a marketing site the same way I treat a change to lead routing. Something upstream of the funnel changed, so I want a check that proves the funnel still works. For a CDN move, that check is simple: can Google still fetch the pages that produce pipeline?

This also connects to AI search. Many AI answer engines rely on their own crawlers, and a blanket bot wall blocks those too. I wrote about how to test that in my guide on checking whether AI crawlers can see your CMS content.

How do you let verified crawlers through without opening the door to scrapers?

Google recommends exempting verified crawlers from bot challenges. Most CDN and firewall providers offer a managed setting for known or verified bots. Google also publishes its crawler IP ranges, so you can confirm that a request claiming to be Googlebot really comes from Google before you let it through.

The verification side is well documented. Google's own documentation describes two methods. The manual method is a reverse DNS lookup on the requesting IP, a check that the hostname ends in googlebot.com, google.com, or googleusercontent.com, and then a forward lookup to confirm it resolves back to the same IP. The automatic method compares the IP against JSON files Google publishes, such as common-crawlers.json for Googlebot and special-crawlers.json for crawlers like AdsBot.

In practice, most marketing teams will never run those lookups by hand. They will rely on the verified bot toggle in their CDN dashboard. My advice is to confirm that toggle is on before cutover, not after, and to write down who owns it. A setting with no owner is a setting that gets changed during the next security review.

If you genuinely need to block specific scrapers, the Google post suggests robots.txt or targeted rules aimed at those bots, rather than a wall that catches everything.

What does pre-warming a CDN cache mean in practice?

Pre-warming means requesting your important pages through the CDN before you send all traffic there, so the edge already holds a copy. Google suggests warming your highest-traffic pages, upcoming promotional pages, and most-crawled URLs at a controlled pace, so the first wave of crawler requests does not all hit your origin.

For a B2B site, I would build the warm-up list from three sources. First, your top landing pages in Google Search Console. Second, the pages your sales team sends most: pricing, security, integrations, and the two or three case studies reps actually use. Third, your sitemap, because that is roughly what crawlers will walk.

The phrase "controlled pace" matters. If you script a warm-up that fires thousands of requests in a minute, you have just recreated the problem you were trying to avoid. A slow loop that works through the list over an hour or two is kinder to your origin.

A cache will not fix everything. If a page is slow because of heavy scripts or oversized images, the CDN only hides part of it. I covered that limit in what a CDN cache will not fix on a slow page.

When should you use a 429 or 503 response to slow Googlebot down?

Only as a short, temporary brake. Google says that if crawling strains your origin during warm-up, you can return 429 or 503 responses, and its crawl rate documentation suggests a window of a couple of hours, or one to two days. Serving those codes on the same URL for multiple days risks that URL being dropped from the index.

This is the advice teams most often get wrong. A 503 feels safe because it means "come back later." It is safe, for a short time. Google's documentation on reducing crawl rate says that serving a significant number of these errors slows crawling across the whole hostname, and that crawling picks back up on its own once the errors stop.

The same documentation mentions a Retry-After header with a 503 or 429, which can state a delay in seconds or a specific time. It also notes that if you cannot serve errors, you can file a special request through the Search Console Googlebot report to lower the crawl rate, though you cannot use that route to ask for more crawling.

My rule is simple. If you are serving 503s to Googlebot on purpose, someone has a calendar reminder to stop.

What should you check in the first week after a CDN cutover?

Check that Googlebot is getting real pages. Use the URL Inspection tool in Google Search Console on your pricing page and top landing pages, watch the Crawl Stats report for spikes in errors, and confirm your firewall logs show verified crawlers passing rather than being challenged.

The Crawl Stats report is the most useful single view here. It shows crawl requests over time and breaks responses down, so a wall of challenges or server errors stands out. If host status warnings appear, I explained what each one means in my piece on crawl stats host status warnings.

I also check one thing that has nothing to do with Google: the forms. Bot protection that challenges crawlers can also break form submissions that post to HubSpot, Zapier, or a webhook, especially if the form handler looks like automated traffic. A CDN migration that blocks Googlebot and leads at the same time is the worst version of this story.

Finally, I compare organic sessions and demo requests by landing page for the two weeks before and after. Not because a dip proves anything, but because a sharp drop on specific pages tells you exactly where to look.

Does this apply if your site runs on a managed host like Webflow?

It applies most when you add your own proxy or firewall in front of a managed host. Managed platforms such as Webflow run their own delivery layer, so the cold cache problem is mostly theirs to solve. The risk returns the moment your team adds a separate CDN or security layer on top.

This setup is more common than you would expect. A security review asks for a web application firewall, so the team puts Cloudflare in front of a Webflow site. That is a reasonable choice. It just means the bot settings in that new layer now decide whether Googlebot reaches your pages.

If that is your situation, the Google post is your checklist. Exempt verified bots, confirm it with URL Inspection, and do not leave challenge pages running against crawlers. For your managed host's own caching and hosting behavior, read the platform's official documentation rather than assuming it matches your proxy.

The bigger point is ownership. When two layers both control delivery, nobody feels responsible for crawling. Pick one person who owns "can Google reach the site" and give them access to both dashboards.

What should you do next?

If you have a CDN migration planned, add three steps to the cutover plan: confirm verified bots are exempt from challenges, pre-warm your pipeline pages, and set a short, dated window for any 429 or 503 brake. If you already migrated, run URL Inspection on your pricing page today.

This is the kind of change I like catching before launch rather than after. It costs one checklist item now, or weeks of lost organic pipeline later. The Google post is clear and short, and I would send it to whoever owns your DNS and firewall settings this week.

If you are planning a migration or a launch and want a second pair of eyes on the site and the systems behind it, from crawling to forms to CRM routing, reach out. I build websites and the automations behind them for lead generation, and this is exactly the kind of check I build into every launch. Let's chat.