How do you audit your crawler control files in an hour?

How do you audit your crawler control files in an hour?

Open your robots.txt, your sitemap, and a sample of your page source side by side, then look for disagreements rather than errors. Most sites have no broken file and plenty of contradictions, and contradictions are what quietly remove pages from search and from AI answers.

I run this audit at the start of most engagements because it is fast, it is boring, and it finds real money. A page blocked by one file while being advertised by another is a page you paid to create and cannot get value from.

Here is the order I use, with the parts of Google's own documentation that decide the answers.

Which files are we actually talking about?

Four things. Your robots.txt, your sitemap or sitemap index, the meta robots and canonical tags inside your pages, and any AI specific file such as llms.txt. Together they tell machines what to fetch, what to list, and which version counts.

They are separate systems with separate jobs, which is exactly why they drift. Robots.txt controls fetching. A sitemap suggests what exists and is worth looking at. Meta robots and canonicals control what happens after a fetch. Nothing in the platform forces them to agree with each other.

The audit is therefore not a syntax check. It is a consistency check, and you can do it by hand on a small site or with a crawler like Screaming Frog on a large one.

What does Google's own documentation say about robots.txt?

More than most people have read. Google enforces a robots.txt file size limit of 500 kibibytes, and content after that maximum file size is ignored. The file must sit in the top-level directory of a site, the URL is case sensitive, and a robots.txt on a subdomain is only valid for that subdomain.

Two more details in that documentation matter for an audit. Path values are relative to the root of the site the file was fetched from, using the same protocol, port number, host, and domain names, and a path must start with a slash and is case sensitive. Field names, on the other hand, are case insensitive, so User-agent and user-agent both work.

The failure behaviour is worth knowing too. Google's documentation says that if it cannot fetch a new version of your robots.txt, for the next 30 days it will use the last good version while still trying to fetch a new one, and that if no cached version is available, Google assumes there are no crawl restrictions. That last sentence is the one to sit with. A robots.txt that fails to serve does not fail closed.

What is the first thing to check, and why is it usually wrong?

Check that the rules you think are live are the rules being served, from the exact host your visitors use. Fetch robots.txt on both the www and non www versions, over https, and compare. Different hosts are different files as far as the rules are concerned.

The second common miss is path case. Because the path value is case sensitive, a rule written for one capitalisation does not cover the other, and CMS platforms are perfectly capable of serving both. If you inherited a site, assume nothing about which URLs a rule actually matches until you have checked a real example.

The third is scope creep. Rules added during a migration to keep a staging area out of the index tend to survive the migration. I have found disallow rules covering entire content directories that nobody could explain, left behind from a launch two years earlier.

What should your sitemap agree with?

Everything in your sitemap should be a URL you want indexed, that returns 200, that is canonical to itself, and that robots.txt allows. Any URL failing one of those four tests is noise, and a sitemap full of noise is a weaker signal than a shorter honest one.

The disagreement I find most often is a sitemap listing URLs that redirect. That happens naturally after a slug change, since the CMS updates the page and the sitemap while the old URL keeps working through a redirect. If both end up listed, you are telling a crawler that two URLs are worth its time and only one is.

Blocked URLs in a sitemap are the more interesting contradiction, because you are simultaneously asking for a URL to be listed and refusing to let it be fetched. Which takes priority is not something to reason about, it is something to remove. On a big site this is where most of the wasted attention lives, and what actually matters in a large sitemap goes deeper on prioritising that cleanup.

Where do canonicals and robots rules contradict each other?

They contradict when you block a URL from being fetched and then expect a signal that lives inside it to be honoured. A canonical tag, a noindex directive, or structured data can only be read if the page is fetched, so blocking the fetch throws away the instruction.

This is the single most expensive misunderstanding in this whole area. Disallowing a URL is not a way to keep it out of an index, it is a way to keep yourself from being able to say anything about it. If your goal is exclusion from search results, the mechanism belongs in the page, and the page has to be reachable for that to work. I wrote about the exact difference between those two tools in noindex versus a robots disallow, and it is the check I would run first on any site with pages you thought were hidden.

Self canonicals deserve a quick pass as well. A template that hard codes one canonical URL across a whole collection is a five minute bug and a six month traffic problem, and it shows up immediately if you spot check three CMS pages rather than one.

Does llms.txt belong in this audit?

Add it if you like, then keep it consistent with everything else, and do not let it become the interesting part. My honest position is that I have not seen a primary vendor commitment that would make me rely on it, so I treat it as a cheap convention rather than a control.

The cost of adding one is close to zero, and there is a genuine benefit in having a plain text summary of your site's important URLs that you maintain deliberately. The risk is priority inversion. I have watched teams spend a week on an AI specific file while a stale disallow rule was blocking their actual product pages.

If you do publish one, check vendor documentation for how any specific engine treats it rather than trusting a blog post, and keep it in step with your sitemap. A file listing URLs you have since removed is worse than no file. The setup itself is straightforward, and the walkthrough for llms.txt on a Webflow site covers it.

What about subdomains, staging, and preview hosts?

Treat every host as its own problem, because robots.txt on a subdomain is only valid for that subdomain and does not cover other subdomains, protocols, or port numbers. A staging host needs its own protection, and the safest protection is authentication rather than a polite request.

Password protecting staging solves the problem completely. A disallow rule asks well behaved crawlers not to look, which leaves unfamiliar crawlers, link discovery, and anyone who shares a URL. If the content must not be public, it should not be publicly fetchable.

Do the same check for any additional hosts that grew around your site. A help centre, a docs subdomain, a careers page on a third party platform, and a marketing microsite each serve their own rules, and nobody audits the ones they forgot they own.

How do you keep this from drifting again?

Write down what each file is for, then check the set whenever you change URLs, launch a section, or migrate a platform. Those three events cause nearly every contradiction I find, and each one takes ten minutes to check at the time instead of an hour to diagnose later.

I keep this as a short list in the project notes for every site I maintain, not because the checks are hard to remember but because the moment to run them is easy to miss. Launch days are busy, and crawler files are invisible until they are expensive.

If you use a CDN or an edge layer in front of your site, add one more note about which layer serves your robots.txt, since a rule set at the edge and a file in your CMS can both exist and only one wins.

What should you do next?

Fetch your robots.txt on both hosts right now and read it out loud. Then take ten URLs from your sitemap and check that each one returns 200, canonicals to itself, and is not blocked. If anything disagrees, fix the disagreement before you write another page.

If your site is large enough that this sounds tedious, that is exactly the kind of audit I do for clients, and the findings usually pay for themselves in the first fix. Reach out if you want someone to run it with fresh eyes.