What Does Google's Crawl Stats Report Tell a Small Site?

Should a small website owner look at Google's Crawl Stats report at all?

Rarely, and Google says so directly. Its help page states that the report "is aimed at advanced users" and that a site with "fewer than a thousand pages" should not need it. Still, a small site can use it for three quick checks: host problems, error spikes, and whether new pages are being discovered.

I bring this up because small business owners and marketers find the report, see charts full of numbers, and assume they have a crawl problem. Most of the time they do not. A typical B2B site with a few dozen service pages and a blog is well within what Google crawls comfortably.

So this piece does two things. It explains what the report shows, in plain terms. And it tells you which parts are worth a monthly glance on a small site, and which parts you can safely ignore.

What is the Crawl Stats report?

The Crawl Stats report is a Search Console view of how Googlebot has been requesting pages and files from your site. It shows how many requests Google made, how your server responded, what types of files were fetched, and why they were fetched. It is diagnostic data about crawling, not about rankings or traffic.

According to Google's Crawl Stats documentation, the report breaks requests down by crawl response, file type, crawl purpose, and Googlebot type. It also shows host status, which tracks whether Google could fetch your robots.txt file, resolve your DNS, and connect to your server.

It is worth separating this from the Page indexing report, which tells you which pages are in Google's index and why others are not. Crawling is a step before indexing. A page can be crawled often and still not be indexed, and the two reports answer different questions.

Why does Google say small sites do not need it?

Google says small sites do not need it because crawl capacity is rarely the bottleneck for them. With fewer than a thousand pages, Googlebot can usually crawl everything that matters without strain. Problems on small sites tend to come from content, internal links, or indexing signals, not from how often Google visits.

The exact wording on Google's help page is: "If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail." That is unusually direct guidance. I take it at face value.

Large sites are a different story. A catalog with hundreds of thousands of URLs, or a CMS that generates endless filter combinations, can have real crawl budget issues. I covered that situation in what matters for crawl budget on a large CMS site. If you run a small Webflow or WordPress site, that article is probably not your problem.

Which part of the report should a small site check first?

Check host status first. It tells you whether Google had trouble fetching your robots.txt file, resolving your domain, or connecting to your server. Any of those can stop crawling across the whole site. On a small site, a host problem matters far more than any change in request counts.

A robots.txt problem is the one I worry about most. Google's robots.txt documentation says that when fetching the file returns server errors, for the first 12 hours "Google stops crawling the site but keeps trying to fetch the robots.txt file." That turns a small configuration issue into a site-wide visibility problem. If host status shows issues, fix them before looking at anything else.

DNS and connectivity problems usually point to hosting or domain configuration. If you recently moved hosts, changed DNS providers, or pointed your domain at a new platform, host status is a good place to confirm the move went cleanly.

What do crawl response codes tell you?

The response breakdown shows how your server answered Google's requests. Mostly you want successful responses. A sudden rise in not found errors usually means broken links or deleted pages. A rise in server errors means your host struggled. On a small site, the trend matters more than any single number.

Not found responses are normal in small amounts. Old URLs linger in Google's memory, and people link to pages that no longer exist. What you watch for is a jump. If not found responses spike right after a redesign or migration, you probably missed some redirects.

Server errors deserve faster attention, because they often mean real visitors see errors too. If you need a page to be unavailable for a while, return the right status code. I explained which one in what a temporarily unavailable page should return.

What does discovery versus refresh mean?

Discovery and refresh describe why Google requested a URL. Google defines discovery as a request for a URL that "was never crawled by Google before," and refresh as "a recrawl of a known page." On a small site, the split tells you whether new pages are being found and existing pages revisited.

If you publish new blog posts or service pages and discovery requests stay near zero for weeks, Google may not be finding them easily. The usual causes are weak internal linking, a sitemap that does not include the new URLs, or pages buried deep in navigation.

Refresh is the larger share on most established sites, and that is normal. Google keeps checking pages it already knows for changes. A healthy small site shows a steady stream of refresh requests with occasional discovery bumps when you publish.

What can a small site safely ignore in this report?

Ignore day-to-day swings in total requests, small changes in average response time, and the split between Googlebot types. These numbers move for reasons you cannot control and rarely affect a small site's search performance. Chasing them wastes time better spent on content and internal links.

Total crawl requests going up or down by a meaningful share in a week is not a sign of trouble on its own. Google adjusts crawling for many reasons. What matters is whether your important pages are indexed and up to date, which you check in other reports.

The file type breakdown is mostly a curiosity on small sites. If you see a large share of requests going to scripts or images, that is usually fine. It becomes interesting only on very large sites where those requests compete with HTML pages for crawl attention.

How often should a small site check the report?

Once a month is plenty, plus once after any migration, redesign, domain change, or hosting move. The monthly check takes a few minutes: look at host status, glance at the response breakdown for spikes, and confirm discovery requests show up after you publish new pages. Then close the report.

The post-change check is where the report earns its keep for a small site. After a migration, host status and response codes tell you quickly whether Google can reach the new setup and whether redirects are working.

Pair it with your sitemap. If discovery looks thin, compare the URLs you think exist with the URLs your sitemap lists. I covered how those drift apart in why your sitemap drifts from your real page set. If you compare Crawl Stats with your own server or CDN logs, first filter out impostors using the method in how to tell a real Googlebot from a fake one.

What should you do next?

Open the Crawl Stats report once this month and check only three things: host status, any spike in not found or server errors, and whether discovery requests appear after you publish. If all three look normal, close the report and spend your time on content, internal links, and indexing instead.

Put a reminder in your calendar to check it again after any site change. That single habit catches most of the crawl problems a small site will ever have.

If you are planning a migration or redesign and want the technical side handled properly, from redirects to crawl checks, reach out. Let's chat about your site.