What Do ETag and Last-Modified Headers Do for Crawling?

Do HTTP caching headers really matter for SEO on a B2B site?

They matter more for large sites than small ones, but every site benefits from getting them right. ETag and Last-Modified headers let a crawler ask "has this page changed?" instead of downloading it again. On a big CMS with thousands of pages, that saves crawl effort for the pages that actually changed.

Most marketers never look at HTTP headers. They are invisible on the page and buried in developer tools. But they shape how search crawlers and other systems spend their time on your site. If your blog has a few thousand posts and your product pages change every week, you want crawlers to notice the changes quickly and skip the pages that stayed the same.

I run into this topic most often with content-heavy B2B sites, the kind with a large blog, a glossary, and many landing pages. The question is usually framed as "why is Google slow to pick up my updates?" Caching headers are one piece of that answer, and they are easy to check.

What are ETag and Last-Modified headers?

They are HTTP response headers that describe a version of a page. Last-Modified gives the date and time the content last changed. ETag gives an identifier, often a hash, that changes whenever the content changes. A crawler stores them and sends them back on its next visit to ask whether anything is new.

The request side has matching headers. When a crawler has a stored Last-Modified date, it can send If-Modified-Since. When it has a stored ETag, it can send If-None-Match. The server compares the values. If nothing changed, it can reply with a short "not modified" response instead of the full page.

That short reply is the HTTP 304 status code. It carries no page body, so it is cheap for both sides. The crawler keeps its stored copy and moves on. If the page did change, the server sends the full content with new header values, and the cycle starts again.

This mechanism is old and widely used across the web, by browsers, content delivery networks, and crawlers alike. It is not an SEO trick. It is basic web plumbing that happens to help search crawling.

How does Google use these headers when crawling?

Google's crawler documentation says its crawlers support ETag with If-None-Match and Last-Modified with If-Modified-Since. When both are present, Google uses the ETag, as the HTTP standard requires. Google recommends ETag because it avoids date formatting problems, and suggests setting both since other systems use them too.

The same documentation adds a few useful details. If you use Last-Modified, Google asks for a clear date format, in the style of "Weekday, DD Mon YYYY HH:MM:SS Timezone." You can optionally set a max-age value in the Cache-Control header to signal how long content is expected to stay the same. Google says other HTTP caching directives are not supported.

Google also notes that caching support varies by crawler and product. Its example is that Storebot-Google only supports caching in certain conditions. So caching headers improve efficiency for many Google crawlers, but you should not assume every Google fetcher treats them the same way.

Finally, Google tells site owners to work with their hosting or CMS provider to implement caching. That is realistic advice. On hosted platforms, you usually cannot set these headers by hand, so the question becomes whether your platform sets them correctly for you.

What happens when a server returns a 304?

A 304 tells Google the content is the same as the last time it was crawled. According to Google's documentation on HTTP status codes, the indexing pipeline may recalculate signals for the URL, but otherwise the status code has no effect on indexing. Your page stays indexed as it was.

This is the part people worry about. A 304 does not drop your page or mark it as stale in a harmful way. It simply means "use what you already have." For a page that genuinely has not changed, that is exactly what you want.

The risk is the opposite case. If your server says "not modified" when the page did change, the crawler keeps the old version. That can happen when a header is computed from the wrong thing, such as a template file instead of the final page, or when a cache layer serves stale headers.

So the goal is not to return as many 304 responses as possible. The goal is to return a 304 only when nothing changed, and a fresh 200 response with new headers when something did. Accuracy beats cleverness here.

Which sites benefit most from correct caching headers?

Large sites with many pages that rarely change benefit most, such as big blogs, documentation hubs, glossaries, and directory-style CMS collections. Crawlers can skip the unchanged pages quickly and spend more effort on new and updated ones. A ten-page brochure site will see little practical difference either way.

Think about a B2B blog with a thousand posts. Most of those posts do not change week to week. Each recrawl that ends in a cheap 304 is effort saved. Meanwhile, the ten posts you updated this month and the new ones you published get fetched in full. That is a sensible use of crawl effort.

I wrote about what actually matters for crawl budget on a large CMS site. Caching headers are one of the supporting pieces. They will not fix weak internal linking or thin pages, but they remove waste from the crawl.

For small sites, I would not spend much time here beyond a quick check. If your platform handles it, great. If not, other work, like clearer pages and better internal links, will matter far more.

How do caching headers relate to sitemap lastmod dates?

They answer the same question in two places. A sitemap lastmod date tells crawlers when a page last changed before they visit. HTTP caching headers answer it during the visit. When both are accurate and agree, crawlers get a consistent signal. When they disagree, you create confusion that slows down discovery of real changes.

The common failure is a lastmod date that changes every day for every page, because the sitemap generator stamps the build time instead of the content change time. Crawlers learn to ignore a date that is always new. Accurate caching headers cannot fully rescue a sitemap that cries wolf.

I covered this in more depth in what a sitemap lastmod date actually does. The principle is the same for both signals. Only report a change when the content a reader sees actually changed.

A good test is to edit one page and leave another alone. After publishing, the edited page should show a new lastmod and new caching header values. The untouched page should show neither. If both pages change, something upstream is stamping everything.

How do you check your own site's caching headers?

Request a page twice and compare. Use your browser's developer tools network tab or a command-line tool like curl to view the response headers. Look for ETag and Last-Modified. Then send the same request with If-None-Match set to that ETag and see whether the server returns a 304.

In the browser, open developer tools, go to the network tab, reload the page, and click the main document request. The response headers section shows what the server sent. Write down the ETag and Last-Modified values if they exist.

With curl, a request that includes only headers will show the same values, and a second request that passes the ETag back in an If-None-Match header shows whether the server honors it. If you get a 304, conditional requests work. If you get a full 200 every time, they do not, at least at that layer.

Check more than one page type. Test a blog post, a CMS collection page, a static landing page, and the homepage. On some setups, different page types sit behind different cache rules, so one page working does not prove they all do.

What if your platform does not set these headers correctly?

Raise it with your hosting or CMS provider first, as Google suggests, since most hosted platforms control headers for you. If you run your own infrastructure or a CDN in front of the site, you can often configure them there. Either way, test after every change, because a wrong header is worse than none.

On a fully hosted builder, your options are usually limited to what the platform exposes. Some platforms let you add custom response headers through a CDN or reverse proxy you put in front. That adds complexity, so I only recommend it for large sites where crawl efficiency is a real, measured problem.

Header behavior also affects how other headers reach crawlers. I wrote about what an X-Robots-Tag header does that a meta tag cannot, and the lesson carries over. Once you put a proxy in front of your site, you own every header it sends, so document what you change.

If you cannot fix it, do not panic. Google can still crawl and index your site without conditional requests. You lose some efficiency, not visibility. Focus on the things you control, like accurate lastmod dates and clear internal links.

What should you do next?

Check a few page types on your site for ETag and Last-Modified headers, then test whether a repeat request returns a 304 when nothing changed. Make sure your sitemap lastmod dates only change when content changes. If headers are missing or wrong on a large site, raise it with your hosting provider before building workarounds.

Treat this as a one-hour audit, not a project. Most sites either pass or have a single upstream fix. The value is knowing which situation you are in, so you stop guessing when updates seem slow to appear in search.

This kind of technical check sits underneath the GTM work I do. I build websites and the automations behind them for lead generation, and a site that gets crawled efficiently is one where new landing pages and updated pricing reach buyers faster.

If you want a second pair of eyes on your site's crawling setup, or help deciding whether caching headers are worth fixing on your platform, reach out. Let's chat about what your headers are telling crawlers today.