What Does an Allow Rule for AI Crawlers Actually Guarantee?

What does an Allow rule for AI crawlers actually guarantee?

That a compliant crawler is permitted to fetch the path. Nothing else. Not that it will visit, not that it will read you, not that anything you publish will be cited, and not that a crawler which ignores the file will behave any differently. Permission and outcome are completely separate things.

This matters because a lot of advice in the last two years has treated robots.txt as a growth lever. Add these Allow lines and get into the answers. My own robots.txt permits universal crawling and carries explicit Allow directives for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, anthropic-ai, Google-Extended and Applebot-Extended, plus a sitemap reference. That was a deliberate choice, and it is not why anything of mine gets cited.

The file is a gate, not an invitation. Opening a gate does not make anyone walk through it. Worth knowing before you spend an afternoon tuning one.

What does the robots specification actually say?

RFC 9309 is short and precise. It states that the most specific match found must be used, and that the most specific match is the one with the most octets. It also states that if no match is found in a group for a matching user agent, or there are no rules in the group, the URI is allowed.

That second point undoes a lot of the folklore. A crawler with no rules addressed to it is already allowed everywhere by default. Adding Allow lines for an agent that has no Disallow anywhere changes nothing about what it is permitted to fetch, because it was permitted already.

The specification also notes that the robots.txt URI is implicitly allowed, which is a small detail with a practical consequence: you cannot hide your robots file from crawlers using your robots file. People do try.

Why does allowing a bot not make it visit?

Because crawling budget is decided by the crawler, not by you. The file describes what a crawler may do. What it actually does depends on whether it knows your site exists, whether it considers the content worth fetching, and how much capacity it wants to spend there. None of that is in your control.

The useful comparison is with traditional search, where the same distinction has always applied and everybody eventually learned it. Being crawlable was never the same as being crawled, and being crawled was never the same as being indexed. AI crawlers have simply added another layer to the same ladder.

So if the question behind your robots.txt edit is why am I not showing up, the file is almost never the answer. I go through the whole set of files worth checking in the piece on auditing your crawler control files, and the honest conclusion there is that the audit is worth doing to find mistakes, not to find growth.

Does allowing a crawler mean you will be cited?

No, and the specification never claimed otherwise. Access is a precondition for citation, not a cause of it. A crawler that is allowed to fetch your page still has to decide your page is the best available answer to somebody's question, in competition with everything else it has fetched.

What allowing does do is remove one way to fail. If a crawler is disallowed, you are definitively out. That is the real argument for permissive rules: not that they win you anything, but that they stop you losing by accident, which is a much smaller and much more defensible claim.

The reasons a particular source gets quoted instead of you are usually about the content and the context rather than the access rules. I went into one version of that in the piece on why ChatGPT cites Reddit rather than my website, and none of the answers there were fixable in a robots file.

Which OpenAI agents does robots.txt actually govern?

Different ones do different jobs, and OpenAI documents the split. OAI-SearchBot is described as being used to surface websites in search results in ChatGPT's search features, and OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. GPTBot is a different question entirely.

OpenAI describes GPTBot as used to make their generative AI foundation models more useful and safe, and states that disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. So the two agents are two separate decisions: whether you want to appear in answers, and whether you want to contribute to training.

That distinction is the single most practically important thing in this whole area, and it is routinely collapsed into one decision by people writing robots files in a hurry. Blocking everything because you do not want to be training data will also remove you from the place where you might have been surfaced. They are not the same lever.

What about agents acting on a person's behalf?

This is where the model of the file breaks down. OpenAI documents ChatGPT-User as covering certain user actions in ChatGPT and Custom GPTs, states it is not used for crawling the web in an automatic fashion and not used to determine whether content may appear in Search, and says robots.txt rules may not apply because the actions are user-initiated.

Read that last part again, because it is the future arriving quietly. Robots.txt was designed for automated crawling. A request made because a specific person asked a specific question is not crawling in that sense, and the protocol was never written with it in mind.

The practical consequence is that your robots file increasingly governs a shrinking share of the non-human traffic reaching your site. It still matters for the crawling that remains. It is simply no longer the whole story, and any strategy built entirely on it is describing an older web.

Is robots.txt any kind of protection?

No, and the specification says so directly. RFC 9309 states that the Robots Exclusion Protocol is not a substitute for valid content security measures. It is a published preference that well-behaved clients choose to honour. There is no enforcement in it anywhere.

If something must not be fetched by anyone, it needs authentication, not a Disallow line. This is old advice and it keeps needing repeating, because a Disallow line feels like a lock and reads like one in a config file. It is closer to a sign on an unlocked door.

There is also a mild irony worth noticing. Disallowing a path publishes the existence of that path to anybody who reads your robots.txt, which is everybody, since the specification makes the file implicitly allowed. Hiding something by naming it in a public file is not hiding it. If you are weighing up whether a separate instruction file helps here, I looked at that question in the piece on the llms.txt standard proposal.

How stale can your robots.txt be in a crawler's cache?

Up to a day, by the specification's own guidance. RFC 9309 says crawlers may cache the fetched robots.txt contents, and should not use the cached version for more than 24 hours unless the file is unreachable. So a change you make now is not necessarily in effect now.

There is a sharper edge case in the same document. It states that crawlers may access any resources when they receive a 400 to 499 response for the robots file. If your robots.txt returns a 404 during a deployment, a crawler that fetches it in that window is permitted to treat your site as fully open.

That is worth designing around if you rely on Disallow rules for anything meaningful. Make sure the file is served reliably, check it after any migration, and do not assume a rule you added this morning is being respected this afternoon.

What should you actually do with your robots file?

Make two deliberate decisions and write them down. Whether you want to be available to answer engines, and whether you are comfortable with your content being used for model training. Then express exactly those two decisions and stop touching the file.

Check it after every migration and every platform change, because that is when robots files get silently replaced with defaults. A blanket Disallow left over from a staging environment is the single most expensive robots.txt mistake I see, and it is invisible until traffic has already gone.

What I would not do is add long lists of Allow lines for every named agent you have heard of. The specification already allows what is not disallowed, so most of those lines are decoration. Mine exist because I wanted the intent to be readable by a human opening the file, not because they change what any crawler may fetch.

What should you do next?

Open your own robots.txt and read it as though somebody else wrote it. Ask what each line is for and whether you could defend it. Most files contain at least one rule nobody can explain, and unexplained rules are how sites accidentally exclude themselves.

Then separate the two decisions in your head if you have not already. Appearing in answers and contributing to training are different choices with different consequences, and the agents involved are documented separately by the vendors themselves. Read the vendor's own page rather than a summary.

If you are trying to work out whether your site is genuinely available to the engines that matter to you, or you have inherited a robots file nobody understands, reach out. It is usually a twenty minute conversation and occasionally it turns up something expensive.