How Do You Check Whether AI Crawlers Can See Your CMS Content?

How do you check whether AI crawlers can see your CMS content?

Fetch the page the way a simple crawler would, without running JavaScript, and look for your CMS text in the raw HTML. If the words are there, crawlers that read HTML can see them. If they only appear after scripts run, some crawlers may miss them. Then confirm robots.txt is not blocking those bots.

Many teams assume that if a page looks right in their browser, every crawler sees the same thing. That is not a safe assumption. Your browser runs JavaScript, loads fonts, opens tabs, and fills in content from scripts. A crawler may do far less.

This matters most for CMS-driven pages, like blog posts, case studies, and integration pages, because that is where most of your answerable content lives. If an AI crawler cannot read that content, it cannot quote it.

Why might an AI crawler miss content you can see?

Crawlers differ in how much of a page they process. Some read the HTML the server sends and stop there. Others render JavaScript, but not always fully or quickly. Content that loads late, sits behind a click, or comes from a third-party script is the most likely to be missed.

Think about how your CMS content reaches the page. If it is written into the HTML when the page is published, any crawler that reads HTML can see it. If a script fetches it after the page loads, a crawler that does not run that script sees an empty container.

The same goes for content hidden behind interactions. Tabs, accordions, and "load more" buttons can all hide text until a user acts. I covered how that affects AI answers in why AI answers skip content inside tabs and accordions. The check in this post is how you find out whether your own pages have the problem.

What is the fastest test you can run?

View the page source, not the inspected page. In most browsers, right-click and choose the option to view source, then search for a sentence from your CMS content. Page source shows the HTML the server sent. If your sentence is there, the content is in the raw HTML.

This is different from the developer tools Elements panel, which shows the page after scripts have run. The Elements panel can show text that was never in the original HTML. For this test, the raw source is what matters.

Pick a distinctive sentence from the middle of the content, not the title. Titles are often in the HTML even when the body is not. If the body sentence is missing from the source, you have found content that depends on scripts.

How do you test the page like a crawler from the command line?

Use a tool like curl to request the page and save the response, then search the output for your sentence. curl does not run JavaScript, so it shows roughly what a simple HTML crawler receives. You can also set a user agent string to mimic a specific crawler for the request.

A typical check looks like requesting the URL, saving the HTML to a file, and searching the file for your sentence. If you want to go further, run the same check with a crawler's user agent set. Some servers and firewalls treat bots differently, so the response a crawler gets may not match the one your browser gets.

That last point matters more than people expect. A content delivery network or bot protection setting can serve a challenge page to unknown bots. If the response to a crawler user agent is a block page or an error, the content is invisible to that crawler no matter how the page is built.

How do you check robots.txt for AI crawlers?

Open yoursite.com/robots.txt and look for rules that name AI crawlers or block all bots from the paths where your CMS content lives. A disallow rule on your blog folder, or a blanket block on unknown bots, will stop compliant crawlers before they ever read the HTML.

Look for named user agents such as GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Each AI company documents its own crawler names and what they are used for, so check their official pages for the current list and purpose. Rules for one bot do not automatically apply to another.

Also check for leftover rules from old projects. A disallow added during a redesign, or a block on a staging path that matches your live CMS path, can quietly hide a whole collection. I wrote about testing these changes safely in how to test a robots.txt change before you publish it.

What about Webflow CMS pages specifically?

For Webflow CMS pages, check the collection template page and a few real items, not just the static pages. Look at content in rich text fields, reference fields, and any sections filled by custom code or third-party embeds. Embedded widgets and script-driven filters are the usual places where content goes missing.

Content you place directly in the CMS and bind to the template is usually the safest to test first. The risk tends to come from additions. A reviews widget, a script that pulls in related posts, a filtering library, or a custom code embed that injects text after load.

Run the source check on three items from each collection. If all three show the body text in the source, the template is probably fine. If any one fails, compare what is different about that item. Often it is a field rendered through an embed rather than a native element.

How do you check whether crawlers are actually visiting?

Look at your server or CDN logs for requests from AI crawler user agents. If your hosting does not expose raw logs, use your CDN's bot analytics or a log-capable proxy. Visits confirm crawlers can reach the pages. The earlier tests confirm what they see once they arrive.

Access to logs depends on your hosting setup. Some platforms give you full logs, some give you summaries, and some give you nothing at the request level. If you run a CDN in front of your site, it is often the easiest place to see bot traffic.

Remember that user agent strings can be faked. If you need certainty that a visit came from a specific company's crawler, check that company's documentation for how to verify its crawler, which often involves IP ranges or reverse DNS lookups.

What should you fix first if content is missing?

Move important content into the HTML that the server sends. That usually means binding it directly to the CMS template instead of loading it with a script, showing key text by default instead of hiding it behind a click, and removing bot blocks that stop crawlers you want. Fix your highest value pages first.

Not everything needs fixing. A related posts widget that loads late is a small loss. A pricing table or the main body of a case study that loads late is a big one. Rank by how often buyers ask questions that page should answer.

After each fix, run the same source check and curl test again. The goal is simple: the sentence you care about appears in the raw HTML, and the crawler you care about gets a normal response.

How often should you repeat this audit?

Run it after any template change, new script, or CDN setting change, and do a full pass each quarter. Sites change in small ways that add up. A single new embed or bot protection rule can hide content across a whole collection without anyone noticing on the live page.

I keep a short list of test URLs for each site, one per collection plus the key static pages. Running the checks on that list takes minutes. It is one of the cheapest ways I know to protect a site's visibility in both search and AI answers.

Write down what you tested and what you found. A dated record makes it easy to spot when something changed and what caused it.

What should you do next?

Pick one CMS item from each collection, view its source, and search for a body sentence. Then fetch the same URL with curl, check your robots.txt for AI crawler rules, and look at your CDN or server logs for crawler visits. Fix any page where key content is missing from the raw HTML.

Repeat after any template or script change. It takes less time than reading this post.

If you want help auditing how AI crawlers and search engines see your Webflow site, reach out. I build websites and the automations behind them for lead generation, and making sure the content can actually be read is where that starts. Let's chat.