Noindex vs. Robots.txt Disallow: Which One Controls AI Crawlers?

Should I use noindex or robots.txt to keep a page out of search?

If you want a page kept out of Google, use a noindex tag, not a robots.txt disallow. Robots.txt stops crawling, but a blocked page can still show up in results. Noindex is the rule that actually removes a page from search. They sound similar, and mixing them up is one of the most common technical mistakes I fix.

This question matters more now than it used to. AI crawlers from OpenAI, Google, and Anthropic have joined the traditional search bots, and they do not all follow the same rules. Knowing which lever does what is the difference between a clean, controlled site and one that leaks pages you meant to hide.

Let me break down what each tool does, where people get burned, and which one you reach for when AI crawlers are the concern.

What is the difference between noindex and robots.txt disallow?

Robots.txt disallow tells a crawler not to visit a page at all. Noindex tells a crawler it may visit, but must not put the page in search results. One controls crawling. The other controls indexing. That single distinction explains almost every confusing thing about how these two tools behave.

Robots.txt sits at the root of your domain and lists paths that named crawlers should skip. It is a request, not a lock, and well-behaved bots honor it. Noindex is different. It lives on the page itself, as a meta robots tag in the head or as an X-Robots-Tag in the server response, and it speaks directly to the indexing step.

Here is the catch that trips people up. For a crawler to obey a noindex rule, it has to read the page first. If robots.txt blocks the page, the crawler never gets in, never sees the noindex, and the rule does nothing. The two tools can cancel each other out.

Why does a page blocked in robots.txt still show up in Google?

A page blocked in robots.txt can still appear in Google because blocking crawling is not the same as blocking indexing. Google explains this directly in its Search Central documentation. If other sites link to your blocked URL, Google can index the address and anchor text without ever visiting the page.

You have probably seen the result. It shows up in search with the URL and a note that says no information is available for this page. Google knows the page exists because of the links pointing at it, but it cannot read the content because robots.txt shut the door. So it indexes a bare, ugly listing instead of a proper one.

Google's own guidance is blunt about this. It says robots.txt is mainly for avoiding overload from crawl requests, and it is not a mechanism for keeping a page out of Google. To keep a page out, you block indexing with noindex or you password-protect it. If you want the deeper mental model, I wrote a full piece on crawling versus indexing for Webflow and AI search.

Which one actually controls AI crawlers?

Robots.txt is the main lever for AI crawlers. Bots like GPTBot, Google-Extended, ClaudeBot, and CCBot are named user agents that read your robots.txt and follow the rules you set for them. Noindex does far less here, because those crawlers are collecting content, not deciding what appears in Google's search results.

This is a real shift in how the web works. The academic research is catching up to it. A 2025 arXiv study on robots.txt gatekeeping found that AI blocking by reputable sites rose from about 23% in September 2023 to nearly 60% by May 2025. Site owners figured out fast that robots.txt is where you draw this line.

News publishers moved hardest. A 2025 BuzzStream study of the top news sites found that 79% now block AI training bots through robots.txt, and 71% block AI retrieval bots as well. If you want to know which of these bots are hitting your own site, I covered that in my post on what an AI crawler is and how to see which ones visit your site.

How do I block AI crawlers like GPTBot and Google-Extended?

You block AI crawlers by adding user-agent rules to your robots.txt file that disallow the specific bots you do not want. You name the crawler, then disallow the paths. Each AI company publishes the user-agent string for its bot, so you target them by name rather than trying to block everything at once.

To keep OpenAI's training crawler out, you add a block for GPTBot. To keep your content out of Google's AI training, you add Google-Extended, which is a robots.txt token that does not affect your normal Google Search ranking at all. For Anthropic, you name ClaudeBot. For Common Crawl, which feeds many AI datasets, you name CCBot. Each gets its own user-agent block.

The important part is that these are separate decisions. Blocking GPTBot from training does not remove you from ChatGPT's live search answers, and blocking Google-Extended does not hurt your Google ranking. You can be picky, allowing the bots that send you traffic while refusing the ones that only take.

Can I use noindex to stop AI from training on my content?

No, noindex is not a reliable way to stop AI training. Noindex is a search directive. It tells search engines to leave a page out of their results. AI training crawlers are not making an indexing decision, so most of them do not treat noindex as an opt-out. For training, robots.txt user-agent rules are the tool that works.

This surprises people who assume one tag covers everything. It does not. Your page can carry a noindex tag, stay out of Google's results, and still be fair game for an AI crawler that only reads robots.txt. If your goal is to keep content out of model training, you have to name the crawlers in robots.txt, not rely on a meta tag meant for search.

There is a newer layer of signals emerging too, like content-signal directives that let you state how your content may be used. I looked at that shift in my write-up on Cloudflare's content signals policy for AI crawlers. For now, robots.txt remains the practical control most bots actually honor.

How do I set each one up in Webflow?

In Webflow, you edit robots.txt under Site Settings in the SEO tab, where you can add user-agent and disallow rules by hand. For noindex, you set it per page in the page settings, or you add a meta robots tag through custom code in the page head. The two live in different places for a reason.

For robots.txt, I paste the AI crawler blocks straight into that SEO field, publish the site, and then check the file at yourdomain.com/robots.txt to confirm it went live. It is a global file, so every rule applies across the whole site. Be careful here, because one stray disallow line can hide pages you wanted found.

For noindex on a single page, Webflow gives you a switch in the page settings to keep it out of search. For a whole template, I use an X-Robots-Tag or a meta robots tag in the custom code. Keep these page-level, so you are not fighting your own robots.txt rules across the site.

When should I use robots.txt disallow instead of noindex?

Use robots.txt disallow when a page should never be crawled at all, like a private script folder, a search results page, or a staging path that wastes crawl budget. Use noindex when a page should be crawled but never listed, like a thank-you page or a thin filter page that adds no search value.

The rule I follow is simple. If the worry is bots wasting resources or reaching things they should not touch, that is a crawling problem, so I reach for robots.txt. If the worry is a page showing up in search where it embarrasses me or splits my rankings, that is an indexing problem, so I reach for noindex.

Never point both at the same page expecting stronger protection. As covered above, a robots.txt block hides the noindex tag from the crawler, so the page can linger in results as a bare link. Pick the tool that matches the actual problem, and use only that one on a given page.

What should you do with your own site?

Start by pulling up your robots.txt and your key pages, and check that no page you want out of Google is only blocked by robots.txt. Move those to noindex. Then decide, crawler by crawler, which AI bots you want to allow and which you want to refuse, and set those rules in robots.txt.

Most sites I audit have this backwards somewhere, with a page hidden the wrong way or an AI crawler no one ever chose to allow. It takes an afternoon to sort out and it gives you real control over both search and AI. If you want a second set of eyes on your robots and indexing setup, reach out. I am happy to walk through your site with you.