How Do You Prove a Crawler Is Really Who It Says It Is?

How do you prove a crawler is really who it says it is?

You check the address it came from, not the name it gave you. Google documents two ways to do this: a manual reverse DNS lookup followed by a forward lookup, or an automatic match of the requesting IP against published lists of Google IP ranges. Everything else is trust without evidence.

This matters more than it used to. Site owners are now making real decisions based on crawler activity, blocking some bots, welcoming others, and in some cases arguing about payment. Every one of those decisions rests on knowing which requests were genuine, and most people have never checked.

What follows is the verification method as the vendors themselves document it, plus the part almost nobody knows: that some legitimate crawlers are documented as ignoring your robots file entirely, by design, and are still behaving correctly.

Why is a user agent string not proof?

Because it is a claim the visitor makes about itself. A user agent is a line of text the requesting software chooses to send. Anyone can send any string they like, which means a log line reading Googlebot is evidence that something wanted to be seen as Googlebot, and nothing more.

Google says exactly why this matters on its verification page, framing it as useful if you are concerned that spammers or other troublemakers are accessing your site while claiming to be from Google. That is the threat model in one sentence, and it comes from the party being impersonated.

The consequences are practical rather than theoretical. If you block by user agent, you block honest bots and miss dishonest ones. If you build an allow rule by user agent, you have effectively published the password. Any rule that depends only on a self-reported name is a rule that can be walked around by editing a string.

So the question is never what did it say it was. The question is where did the request actually come from, and does that origin belong to the organisation being named.

How does Google say to verify its own crawlers?

Two methods, and the manual one takes about thirty seconds. Google's documentation says to run a reverse DNS lookup on the accessing IP address from your logs using the host command, then verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.

Then you do it in reverse. Google's instructions say to run a forward DNS lookup on the domain name you just retrieved, again with the host command, and verify that it resolves to the same IP address that appeared in your logs. Both directions have to agree.

That second step is the entire security of the technique, and it is the step people skip. A reverse lookup alone can be manipulated by whoever controls the reverse DNS for an address range. Requiring the forward lookup to resolve back to the same address closes that gap, because it must be published by the domain owner.

For anything beyond a one-off check, Google describes an automatic route: matching the crawler's IP address against published lists of Google crawler and fetcher IP ranges, in JSON files split by crawler category. Its documentation notes that the IP addresses in those files are represented in CIDR format, which is worth knowing before you try to compare them to a plain address.

What do Google's three crawler categories tell you about robots.txt?

That obedience to your robots file depends on which category a request belongs to, and one category is documented as ignoring it entirely. This is the single most useful thing on that page and almost nobody has read it.

Google's documentation splits its crawlers and fetchers into three types. Common crawlers, the ones used for Google's products such as Googlebot, are described as always respecting robots.txt rules for automatic crawls. Special-case crawlers, such as AdsBot, perform specific functions and are described as crawlers that may or may not respect robots.txt rules. User-triggered fetchers are tools and product functions where the end user triggers a fetch, and Google states plainly that because the fetch was requested by a user, these fetchers ignore robots.txt rules.

Read that last one carefully, because it reframes a whole category of complaint. A request that ignored your robots file is not automatically a rogue request. If a person clicked something and a fetch happened on their behalf, the documented behaviour is that your robots directives do not apply to it.

Each category has its own reverse DNS pattern and its own published IP list, which is how you tell them apart in practice. Common crawlers resolve to googlebot.com addresses. Special-case crawlers resolve to a rate-limited-proxy hostname on google.com. User-triggered fetchers resolve to google.com or to gae.googleusercontent.com, the latter covering fetches originating from sites running on Google Cloud.

Do the AI companies publish the same kind of list?

Several do, and the ones that do are the ones you can actually verify. This is the most useful sorting criterion available right now, because publishing verifiable addresses is a choice a company makes deliberately.

Perplexity's documentation describes PerplexityBot as designed to surface and link websites in search results on Perplexity, and Perplexity-User as supporting user actions within Perplexity, and it publishes IP address endpoints for both. OpenAI documents OAI-SearchBot for surfacing sites in ChatGPT's search features, GPTBot for training its generative AI foundation models, and ChatGPT-User for certain user actions, alongside published address files.

Notice the shape repeating across all three companies. There is a search crawler, a training crawler, and a user-triggered fetcher, and they are separable. That separation is what lets you make a real decision rather than a blanket one, and it only works if you can confirm which one actually visited.

My honest position is that I treat publication of verifiable addresses as a minimum standard of good faith. A company that names its crawler and publishes where it comes from is making itself accountable. One that does neither is asking for trust it has not offered any way to check.

What can you actually see on Webflow hosting?

Less than you would like, which is worth saying plainly rather than pretending otherwise. Webflow's Analyze API documents a traffic report that returns a daily time series of a single metric, sessions, users, or pageviews, over a time window, and it states that the endpoint requires a workspace with the Analyze add-on.

Those metrics describe visits, not crawler identity. A session count does not tell you which user agent arrived, let alone which IP it came from, so it is the wrong instrument for this job. It is a good instrument for a different one, and confusing the two is how people end up arguing about bot traffic using numbers that were never measuring bots.

Webflow's documentation also notes an operational limit worth knowing if you plan to automate anything against it: each access token can have one Analyze request in flight at a time across all Analyze endpoints, and additional concurrent requests return a 429 response. Build your polling accordingly.

The practical consequence is that crawler verification on a hosted platform usually has to happen in front of the platform rather than inside it. If you need request-level evidence, you need something sitting between the internet and your site that keeps it.

How does a CDN change what you can observe?

It moves the evidence to where you can read it. A content delivery network sees every request before your host does, which makes it the natural place to identify, classify, and act on crawler traffic.

Cloudflare's documentation for AI Crawl Control, which it notes was formerly called AI Audit, describes exactly this set of jobs. It says the product gives you visibility into which AI services are accessing your content, lets you monitor a dashboard for crawler activity and request patterns, set allow or block rules for individual crawlers, and track which crawlers follow your directives so you can create enforcement rules. Its documentation states it is available on all plans and works with zero configuration. It also mentions pay per crawl pricing for content access, described as a private beta.

The classification model behind it is the part I find genuinely useful. Cloudflare's bot documentation says that rather than relying on a single AI bot label, it classifies bots by behaviour, and it names three AI-related use cases customers can manage directly: search, meaning collecting or indexing your content so it can answer questions about it later; agent, meaning automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents; and training, meaning crawling your content to train or fine-tune a model. It also notes that a single bot can have more than one behaviour.

That last clause deserves emphasis. If one bot can do more than one thing, then a blanket allow or block decision is always approximately wrong. Behaviour-level control is the only kind that matches how these systems actually work, and it is why the old instinct of maintaining a list of bad user agents has stopped being useful. I covered the news side of crawler verification separately in Cloudflare's move on AI crawler verification.

What should you do about a crawler that ignores your rules?

First establish which of three things actually happened, because the correct response is different for each and people routinely pick the wrong one.

The first possibility is that the request was never from who it claimed. Verify it by address. If verification fails, this is not a crawler policy question at all, it is ordinary unwanted traffic, and it belongs in whatever you already use to handle that.

The second possibility is that it was genuine and it belongs to a category documented as not bound by robots directives, such as a user-triggered fetch. Getting angry about this is getting angry at documented behaviour. If you do not want those requests, robots.txt was never the control, and you need an access rule rather than a directive.

The third possibility is that it was genuine, it was an automatic crawl, and it ignored your rules anyway. That is the only case that is actually a violation, and it is worth documenting carefully with the verified address, the timestamp, and the path before you say anything publicly. I would not name a company over unverified log lines, and neither should you. Analytics distortion from bot traffic is a related and much more common problem, which I have written about in AI bots skewing site analytics.

What should you do next?

Take one suspicious line from whatever request data you can reach and verify it properly. Reverse lookup the address, confirm the hostname belongs to the organisation being claimed, then forward lookup that hostname and confirm it resolves back to the same address. If both directions agree, it was real. If not, it never was.

Then decide where your evidence is going to live. If you are on a hosted platform with no request-level logging, put a layer in front that keeps it, and configure your rules there rather than in a file that only well-behaved automatic crawlers are documented to obey. Doing this before you need it is considerably easier than reconstructing it afterwards, and if you want a broader primer, I have written about what an AI crawler is and how to see which ones visit.

If you want a second opinion on your crawler setup, or you have log lines you cannot interpret, reach out. This is usually a short, concrete conversation and it tends to end with fewer rules rather than more.