Should You Block AI Crawlers From Your Pricing Page?

Should you block AI crawlers from your pricing page?

For most B2B software and services companies, no. Blocking AI search crawlers from your pricing page does not stop buyers from asking assistants about your prices. It just means the answer comes from older pages, review sites, or guesses instead of your own page. If you block anything, block training crawlers, not search crawlers.

The question usually comes from a reasonable worry. Pricing changes, discounts are negotiated, and nobody wants ChatGPT quoting last year's numbers to a prospect. Some teams also worry that competitors will use AI tools to scrape and undercut them. Those concerns are real. Blocking the page is usually the wrong fix for them.

My own site takes the opposite approach. The robots.txt on pravinkumar.co explicitly allows GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, and Google-Extended. I want answer engines to read my pages directly rather than piecing my work together from other sources.

What is the difference between AI search crawlers and AI training crawlers?

Search crawlers fetch pages so an AI product can show and cite them in answers. Training crawlers collect content that may be used to train future models. Several AI companies let you control the two separately, so you can stay visible in AI answers while opting out of training if you prefer.

OpenAI's crawler documentation is a clear example. It says OpenAI uses OAI-SearchBot and GPTBot robots.txt tags, and that each setting is independent. It gives the example of allowing OAI-SearchBot to appear in search results while disallowing GPTBot to indicate content should not be used for training its generative AI foundation models.

Google works differently. Its documentation on AI features says robots.txt directives for Googlebot are the control for how sites are crawled for Search, which includes AI Overviews and AI Mode. Google-Extended is the separate control for limiting training and grounding in some of Google's other systems. Blocking Googlebot from your pricing page would remove it from regular search as well.

What happens if you block search crawlers from pricing?

Assistants lose your official source and fall back on whatever else they can find. That might be a three-year-old blog post, a review site listing, a reseller page, or a forum thread. The buyer still gets an answer about your pricing. It is just less likely to be right, and you have no control over it.

This is the core problem with blocking. You cannot remove your pricing from the conversation, because buyers will keep asking. You can only remove your own version of it. For a company that sells on clarity and trust, that is a bad trade.

It can also hurt beyond pricing questions. When someone asks an assistant to compare you with a competitor, price is usually part of the comparison. If the competitor's pricing page is readable and yours is blocked, the answer will describe their pricing with confidence and yours with a guess.

What about ChatGPT-User and other on-demand fetchers?

These fetch a page when a person asks the assistant to look at it, and robots.txt may not stop them. OpenAI's documentation says ChatGPT-User is used for certain user actions and is not used for crawling the web in an automatic fashion. It also says that because these actions are user-initiated, robots.txt rules may not apply.

That detail matters for pricing pages specifically. A buyer who pastes your pricing URL into ChatGPT and asks "what would this cost for 20 seats" may get the page read on their behalf regardless of your crawl rules. Blocking automated crawlers does not close that path, so it is not a reliable way to keep pricing out of AI tools.

The better lever is the page itself. If a person or an assistant reads it, what should they understand? That question leads to page design and copy changes that actually protect you.

When does blocking a pricing page make sense?

When the page shows prices that are wrong for most visitors, such as partner-only rates, legacy plans, or region-specific prices with no clear labeling. In those cases, the real fix is the page, but a temporary block can stop misinformation while you fix it. Treat any block as short term.

Some companies also run gated pricing calculators or customer-only price sheets. Those should not be public pages at all. Put them behind login, where crawlers cannot reach them, rather than relying on robots.txt to hide a public URL.

If your concern is training only, block the training crawler and keep the search crawler. That keeps your pages eligible for citation in AI answers while stating a clear preference about model training. It is a cleaner position than blocking everything and then wondering why assistants misquote you.

How do you make a pricing page that AI assistants read correctly?

Write plan names, prices, billing period, and what changes between plans as plain text on the page, not only inside images, toggles, or calculators. Add a short line on what is negotiated, such as enterprise terms. A clear, readable page is the best protection against assistants quoting the wrong number.

Toggles between monthly and annual pricing are a common trap. If only one state is in the initial HTML, an assistant may read and repeat only that one. Label both clearly in text somewhere on the page, such as "billed annually" next to the number. I covered this pattern in more depth in the article on how assistants handle a pricing toggle.

If your prices are not public, say so plainly and describe what drives the price. "Pricing depends on seats and usage; most teams start between X and Y" is far better than a blank "contact us," as long as the range is accurate. I wrote about that case in how AI assistants handle contact-sales pricing.

How do you check what AI engines say about your pricing today?

Ask them. Put the same pricing questions a buyer would ask into ChatGPT, Perplexity, Claude, and Google AI Mode, and record the answers and cited sources. Do it monthly and after every pricing change. The gap between your page and their answers is your to-do list.

Look at the cited sources, not just the answer. If assistants cite your pricing page and still get it wrong, the page is unclear. If they cite third-party sites, those sites are outranking your own page as a source, and you may need to update listings or contact the publisher.

I laid out a simple monthly routine for this in how to check whether AI engines quote your prices correctly. Pair it with your crawler settings so you know whether a wrong answer is a crawl problem or a page problem.

What should your robots.txt say about pricing, then?

Usually nothing special. Let search crawlers reach the pricing page the same way they reach the rest of your site, and decide on training crawlers as a site-wide policy, not a pricing-page exception. A single, consistent policy is easier to maintain and easier to explain to your team.

Before you change anything, write down your intent in one sentence per crawler type. For example: "Allow AI search crawlers everywhere. Disallow training crawlers site-wide." Then match the file to the sentence. Teams get into trouble when robots.txt grows rule by rule with no stated intent behind it.

If you need a refresher on how the two crawler types differ in practice, see the difference between a training crawler and an answer crawler. And remember that robots.txt states a preference. It is not access control for anything truly private.

What should you do next?

Check your robots.txt for any rule that blocks search crawlers from pricing, and remove it unless the page shows misleading prices. Decide training crawler policy site-wide. Then rewrite the pricing page so plans, prices, and billing periods are clear in plain text, and test what assistants say every month.

Your pricing will be discussed in AI answers whether you allow it or not. The only real choice is whether those answers start from your page or from someone else's version of it.

If you want help reviewing your crawler settings and pricing page for AI search, reach out and let's chat.