Google's Shared AI Training Control: Should B2B Sites Change robots.txt?

Should you change your robots.txt now that Google plans a shared AI training control?

Not yet, but you should plan for it. On October 30, 2026, Google said it plans to support a shared AI training control in robots.txt, aiming for 2027. Nothing changes on your site today. What changes is the decision you will soon make once, for many AI providers at the same time.

I read the announcement on the Google Search Central Blog the way I read any platform change: what does it actually say, what does it leave open, and what should a B2B team do this quarter. The honest answer is that this is a planning item, not an emergency. But it is a planning item that touches your pipeline, because it touches whether AI tools learn from your pages.

Most of the teams I work with never made a conscious choice about AI training. Their robots.txt grew by copy and paste. This announcement is a good excuse to make that choice on purpose.

What exactly did Google announce on October 30, 2026?

Google announced plans to support a shared AI training control in robots.txt. Website owners will be able to use one shared policy token to state a training preference. Google says the work is based on the AIPREF working group at the IETF, and it aims to support the control with other AI providers in 2027.

The post is titled "Supporting a shared AI training control for websites" and it sits on the Google Search Central Blog. It frames the new control as a joint signal rather than a Google-only switch. Google also says that other AI providers can handle the token in a similar way, but the post does not name any other company that has committed to it.

That last detail matters. A shared control is only as useful as the number of providers who honor it. Today you have a stated intent from Google and a standards effort at the Internet Engineering Task Force. You do not yet have a list of providers, and the post does not publish the exact syntax of the token.

How does the new control relate to Google-Extended?

Google-Extended is the training control Google has offered since 2023, and it stays. Google says existing company-specific controls will keep working alongside the new joint signal. If you use the shared token to opt out and do not mention Google-Extended, Google plans to read that as if you had disallowed Google-Extended.

According to Google's crawler documentation, Google-Extended is a standalone product token. It lets publishers manage whether content Google crawls from their sites may be used for training future generations of Gemini models. The same documentation says it does not affect a site's inclusion in Google Search and is not used as a ranking signal there.

So the new control is not a replacement. It is a shortcut that sits on top. If you have already written a Google-Extended rule, the specific rule still speaks for itself. The shared token becomes the default for any provider that supports it and that you have not addressed by name.

Why should a B2B marketing team care about AI training at all?

Because training shapes what AI tools already believe about your category before anyone searches. If a model learned from your pages, it may describe your product in your words. If it never saw them, it learns your category from competitors, review sites, and forum threads. That is a positioning decision, not only a legal one.

I build go-to-market systems for B2B software companies, and the top of every funnel I touch now includes AI assistants. Buyers ask ChatGPT, Gemini, Claude, and Perplexity to explain a category, shortlist vendors, or compare two tools. Some of what those tools say comes from live retrieval. Some of it comes from what the model absorbed in training.

I keep those two separate on purpose. I wrote about the difference between a training crawler and an answer crawler because teams often block one while thinking they blocked the other. The new shared token is about training. It is not, based on what Google has published, a switch for live answers.

Does opting out of AI training hurt your visibility in Gemini answers?

It can, depending on how you set it up. Google says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. That wording ties Gemini grounding visibility to allowing Google-Extended, so a blanket opt-out deserves a real decision, not a reflex.

Read that sentence slowly, because it is the most commercial line in the whole post. Google is telling you that the training control and some answer visibility in Gemini Apps are connected. If your pages could earn clicks from Gemini, a broad opt-out may cost you more than it protects.

My view is simple. For most B2B software companies, public marketing pages exist to be read, quoted, and repeated. Your homepage, product pages, comparison pages, and pricing explanations are written to persuade. Hiding them from the systems buyers now use to research feels like printing a brochure and then locking it in a drawer.

Which pages should you actually keep out of AI training?

Keep out pages where reuse would hurt you more than help you: gated research you sell or trade for leads, customer-only documentation, proprietary benchmarks, and drafts or staging paths. Keep in the pages written to persuade. The test is whether you would be happy to see a paragraph repeated by an AI assistant without a click.

Here is the framework I use with clients. I sort every section of a site into three groups. The first group is persuasion content, which is meant to spread. The second group is value content, such as original research or templates, where the content itself is the thing you trade for a lead. The third group is private or messy content that should not be public to begin with.

Persuasion content stays open. Value content gets a case-by-case decision, often a split where the summary is public and the full asset sits behind a form. Private content should be fixed at the source with authentication or removal, because robots.txt is a request, not a lock. I covered a narrower version of this in whether you should block AI crawlers from your pricing page, and the same reasoning scales to the whole site.

What should you do this quarter while the standard is still being written?

Audit what your robots.txt says today, write down your training policy in one sentence, and decide which provider-specific rules you would keep once a shared token exists. Do not invent syntax ahead of the standard. Wait for the AIPREF work and Google's documentation to publish the exact token before you edit anything.

The audit takes less than an hour. Open your live robots.txt file and list every user agent it names. For each one, note whether the rule is there because someone decided it or because it was pasted from a template. You will often find rules for crawlers nobody on the team can explain.

Then write the policy sentence. Something like: "Our public marketing pages may be used for AI training and answers; our gated research and customer docs may not." That sentence is what you will translate into the shared token later. Having it written now means the change takes ten minutes when the syntax lands, not a debate in a meeting.

When you do edit, test first. A single typo in robots.txt can block far more than you meant to. I walk through that process in how to test a robots.txt change before you publish it, and I would run that check every time you touch the file.

What does this announcement not tell you yet?

It does not give the exact token syntax, a list of committed providers, or a firm launch date beyond the 2027 aim. It also does not explicitly address Google Search or AI Overviews. Treat anything you read elsewhere that fills those gaps with confidence as speculation until Google or the IETF publishes it.

This is where I see teams get burned. A platform posts a plan, and within a day there are confident threads describing how it will work, with example code. Some of those examples will be wrong. If you paste speculative syntax into a production robots.txt, you are guessing on the one file that controls how every crawler treats your site.

So my rule is to separate what is announced from what is assumed. Announced: a shared policy token in robots.txt, based on IETF AIPREF work, with Google aiming for 2027 support, and Google-Extended continuing to work. Assumed by others, and unconfirmed: everything else.

How should GTM teams think about AI training as a channel decision?

Treat it like any channel decision: who you want to reach, what it costs, and what you lose by opting out. AI assistants are now a research layer between your buyer and your website. Your training policy is part of your distribution strategy, so the revenue team should own it alongside legal, not leave it to whoever edits robots.txt.

In most companies I work with, robots.txt belongs to nobody. A developer set it up at launch, an SEO consultant changed a line years ago, and nobody has looked since. That was fine when the only reader was a search crawler. It is not fine when the file quietly decides whether AI tools learn your positioning.

I would put the training policy in the same document as your ideal customer profile and channel plan. It is a statement about where you want your story to travel. Review it once a quarter, the same way you review which channels deserve budget.

What should you do next?

Run the robots.txt audit this week, write your one-sentence training policy, and set a reminder to check the Google Search Central Blog and IETF AIPREF updates. When the token syntax is published, apply your policy, test it, and log the change. That is the whole job, done calmly and once.

Across 100+ projects for 25+ clients, the pattern I trust most is boring: decide the policy before the tooling forces you to. The shared AI training control is a good change for site owners because it reduces the number of rules you need to maintain. It only helps if you know what you want it to say.

If you want a second pair of eyes on your robots.txt, or help turning your AI visibility into an actual part of your go-to-market plan, reach out through pravinkumar.co. Let's chat.