Caching & Cache Headers: How They Affect Crawling

When the Web and Reality Diverge

Every time a browser or a search engine crawler requests a page, there is a question running silently in the background: should the server generate a fresh response, or can an older copy be served instead? That question, and the rules that answer it, sit at the heart of web caching. For most users, caching is invisible and beneficial. For search engines, it creates a timing problem that shapes how accurately a crawler's picture of a site reflects what is actually live.

This lesson explains what caching is, how cache headers communicate freshness rules to crawlers, and why a gap can open between the content a server has today and the content a search engine has on record.

What Caching Actually Is

Caching is the practice of storing a copy of a resource so that future requests for that resource can be served from the copy rather than by regenerating the original. The copy might live in a browser, in a content delivery network (CDN), in a reverse proxy, or in any number of intermediate systems between the origin server and the end requester.

The motivation is efficiency. Generating a dynamic page from a database on every single request is expensive. If the page is unlikely to change in the next hour, serving a stored copy to a thousand visitors costs a fraction of the compute that regenerating it a thousand times would. Caching therefore reduces server load, speeds up response times, and lowers infrastructure costs.

The trade-off is freshness. A stored copy is, by definition, a snapshot from the past. The moment the original resource changes, every cached copy of it becomes stale to some degree. How stale, and for how long, depends on the instructions built into the cache headers.

Cache Headers: The Instructions Servers Send

When a server responds to a request, it can attach HTTP headers that tell any caching layer how to handle the response. These headers are the language servers use to communicate freshness rules. Understanding what they express is essential to understanding why crawlers sometimes see outdated content.

Cache-Control and Max-Age

The Cache-Control header is the primary mechanism for expressing caching policy. Among its many directives, max-age is the most consequential for freshness. A value of max-age=3600 tells any cache that this response is considered fresh for 3,600 seconds (one hour). During that window, a caching layer is permitted to serve the stored copy without contacting the origin server at all.

From a crawler's perspective, this means that if a CDN or reverse proxy sits between the crawler and the origin, the crawler may receive a copy that is up to an hour old without either party being aware that the content has changed. The server considers the cached response valid. The crawler receives a complete, well-formed response. Neither signals that a discrepancy exists.

Expires

An older mechanism, the Expires header, achieves a similar outcome by specifying an absolute date and time after which the cached response should be considered stale. When Cache-Control and Expires are both present, Cache-Control takes precedence in modern systems. The Expires header is less commonly relied upon now but still appears in many legacy configurations, and its presence can extend the window during which stale content is served.

ETag and Last-Modified

These two headers serve a different purpose. Rather than telling a cache how long to keep a copy, they provide a way to validate whether a cached copy is still current. An ETag is a unique identifier tied to a specific version of a resource. A Last-Modified value records when the resource was last changed at the origin.

When a cache wants to check whether its stored copy is still valid, it can send a conditional request to the origin server, including the ETag or Last-Modified value it holds. If the origin confirms nothing has changed, it returns a lightweight 304 Not Modified response, and the cache continues serving its stored copy. If the resource has changed, the origin sends the new version.

This validation mechanism is more efficient than always fetching a full response. But it only runs when the cache decides to check. During a max-age window, the cache may not check at all.

The Crawl Gap: How It Opens

Search engine crawlers are not fundamentally different from browsers in how they interact with caching infrastructure. When Googlebot or any other crawler requests a URL, it receives whatever the caching layer decides to serve. If a CDN has a fresh-enough cached copy, it serves that copy. The crawler processes it, indexes the content, and moves on.

The gap opens when content changes at the origin but the cached copy has not yet expired. Consider a scenario: a page is cached with a max-age of 24 hours. A significant change is made to that page at the origin. A crawler visits six hours later. The CDN serves the cached copy from before the change. The crawler indexes the old content. From the search engine's perspective, the page still contains the pre-change version.

This gap is not a malfunction. Every system involved behaved exactly as designed. The caching layer followed its instructions. The crawler processed the response it received. The problem is that the instructions created a window during which the indexed version of the page diverged from the live version.

Why Search Engines Cannot Simply Bypass Caches

It might seem logical for search engine crawlers to always request fresh content by bypassing caches. Some crawlers do send headers that signal a preference for fresh responses, but this is not a reliable solution for several reasons.

First, caching infrastructure is not always under the control of the site owner in a granular way. A CDN may serve cached content regardless of request headers, depending on how it is configured. Second, forcing fresh responses on every crawl request would impose significant load on origin servers, particularly for large sites. Search engines are sensitive to the burden their crawlers place on servers, and bypassing caches universally would undermine that consideration. Third, the relationship between a crawler and a cache is governed by the headers the server sends, not by what the crawler prefers.

The result is that the crawl budget and cache configuration interact in ways that are not always visible to site owners. A crawler may visit a URL frequently, but if it consistently receives a cached response, the indexed content may lag behind the live content by the duration of the cache's freshness window.

Layers of Caching That Affect Crawlers

Caching does not happen in a single location. Multiple layers can each introduce their own freshness windows, and a crawler's request may pass through several of them.

CDN Edge Caches

Content delivery networks store copies of responses at geographically distributed edge nodes. When a crawler requests a URL, it often reaches an edge node rather than the origin server. The edge node serves its cached copy if one exists and is considered fresh. The freshness rules at the edge are typically governed by the same Cache-Control headers the origin sends, but CDN configurations can override or extend these.

Reverse Proxies

A reverse proxy sitting in front of an origin server can cache responses before they ever reach a CDN. This adds another layer where a stale copy might be served. The proxy's behavior is governed by its own configuration, which may or may not align precisely with the origin's cache headers.

Application-Level Caching

Many web applications cache rendered HTML at the application layer, storing pre-built versions of pages to avoid database queries. When a page changes in the database, the application cache may continue serving the old rendered version until it is explicitly cleared or until a configured expiry time passes. A crawler requesting the page receives the application's cached output, not a freshly rendered version.

The Relationship Between Caching and Crawl Timing

Search engines do not crawl every page on a site continuously. They allocate crawl capacity based on signals including a site's authority, the frequency of observed changes, and server response characteristics. This means there is already a natural gap between when content changes and when a crawler next visits.

Caching can compound this gap. If a crawler visits a page, receives a cached response reflecting old content, and then does not return for several days, the indexed version may be two layers behind reality: the cache served content from before the most recent change, and the crawler has not yet returned to see even the cached version update.

Understanding this layered timing helps explain why indexed content sometimes lags behind live changes in ways that seem disproportionate to the age of the change. The delay is not always about crawl frequency alone. It is often about what the crawler received when it did visit.

What This Means for Understanding Technical SEO

Caching is a fundamental property of how the web delivers content at scale. It exists because the alternative, regenerating every response from scratch for every request, is not sustainable at the volumes modern websites handle. The freshness trade-off is a deliberate design choice, not an oversight.

For anyone seeking to understand technical SEO at a deeper level, recognizing that search engines operate on the web as it actually behaves, including its caching infrastructure, changes how the relationship between a live site and its indexed representation is understood. The indexed version is always a historical record. How historical depends on the interaction between crawl frequency and cache freshness windows.

This understanding reframes common observations. When a change made to a page does not appear in search results for longer than expected, the explanation may not lie in crawl frequency alone. It may lie in what the crawler saw when it visited, which is shaped as much by cache configuration as by the content itself.