Skip to content

ETag and Last-Modified: checking content freshness correctly

Test conditional requests without confusing a 304 response with missing content. Check validators after edits and identify stale page representations.

By Crawlito · 5 min read

In short

  • A 304 response belongs to a conditional request and normally reuses a previously stored representation.
  • A freshness test needs a baseline body, its validator, and a repeat request after a controlled edit.
  • Sitemap dates, visible update dates, and HTTP validators describe different parts of publication.
On this page

ETag and Last-Modified help HTTP clients check whether a stored representation has changed. For a content audit, the useful test is not simply whether those headers exist: capture a page, record its validator, make an authorized edit, and check that the old validator no longer causes the old content to be reused. A 304 response by itself is not evidence of an empty or broken page.

This is a delivery test, not a technique for forcing AI systems to revisit a URL. It helps you avoid serving contradictory freshness signals when public information changes. Run it on a controlled test page or during an ordinary approved publication, rather than changing commercial facts solely for an experiment.

Start with a stored page and a validator

HTTP caching guidance from MDN explains revalidation using validators such as ETag and Last-Modified. A conditional request can receive 304 when the stored representation remains valid, instead of transferring the content again. That response makes sense only alongside the client's existing copy.

Capture the initial public page with GET:

bash
curl --silent --show-error \
  --dump-header baseline-headers.txt \
  --output baseline.html \
  'https://www.example.com/guides/retention'

Check that this is the final canonical address and the response contains the expected guide. If it redirects, resolve that first and repeat the capture at the intended destination. Inspect the header file for ETag and Last-Modified, preserving the exact value including quotes where present.

The baseline matters because a later 304 does not include a replacement body. Without the stored file, you cannot say what the client would reuse. Keep the baseline response, its timestamp, and a short content marker together in your evidence folder.

Also record which public variant you tested. A language, currency, or authentication variant may not be the same representation as the default anonymous page. Reusing a validator while changing those conditions can produce a confusing experiment.

Send one controlled conditional request

If the baseline contains an ETag, repeat the request using that exact value. The value below is illustrative; replace it with the observed value:

bash
curl --silent --show-error \
  --header 'If-None-Match: "observed-revision-value"' \
  --dump-header conditional-headers.txt \
  --output conditional-body.txt \
  'https://www.example.com/guides/retention'

Record the status and any new validator. An unchanged public representation may produce 304. A server may also return the full representation instead. The test's objective is truthful content delivery, not maximizing the number of 304 responses.

If you are testing Last-Modified separately, use the observed HTTP date with If-Modified-Since. Keep the experiment simple by testing one conditional mechanism at a time. Do not invent a future date or use the sitemap's date as though it were an HTTP validator.

Use HEAD versus GET testing if your existing monitoring checks only response headers. For this exercise, the baseline GET body is essential evidence.

Repeat after a meaningful content edit

In a hypothetical documentation page, a retention period changes from 14 to 30 days. Once the normal publishing workflow completes, send the conditional request again using the old ETag. Then make a plain GET request and compare the returned content with the expected revision.

The important failure is not a particular header format. It is an old representation being treated as current after the public document should have changed. If an old ETag still produces 304 while the origin now has different content, investigate validator generation and the intermediary caches before reporting success.

Use a compact results table:

StageRequestEvidence required
Before editOrdinary GETBaseline text and validator
Before editConditional GETStatus and validator behavior
After publicationConditional GET with old validatorWhether changed content is retrieved
After publicationOrdinary GETExpected new public text

This table prevents a common mistake: checking only the editor's new revision while ignoring the response delivered through the public cache. A valid experiment compares the same URL and request conditions across the publication boundary.

Trace incorrect validators to their owner

A validator may be generated by your application server, static hosting system, reverse proxy, or CDN. Find the component that actually writes the public header. Editing an application setting has no effect if a later layer overwrites it.

Look for content-dependent revisions that were replaced with a constant build value, or metadata based on a file whose modification time does not follow the actual page content. These are investigation hypotheses, not conclusions from the mere presence of an ETag. Confirm them by tracing one revision from source to public output.

For an application with multiple dependencies, list what contributes to the representation: article text, shared pricing data, translation files, and any embedded public policy copy. A change to one of those dependencies should be considered in the page's regeneration or validation strategy. Otherwise an unchanged article file can conceal a changed underlying fact.

Coordinate this with the CDN freshness workflow. Purging an edge cache can temporarily expose the right version, but the durable repair is correct generation and invalidation for the next change.

Keep publication dates and HTTP freshness distinct

A visible “updated” date tells a reader when editorial work occurred. Sitemap lastmod communicates a content modification signal in a URL inventory. HTTP validators help clients compare a stored representation with the server's current one. They should be truthful, but they are not interchangeable fields.

Do not change every article's editorial date simply because assets were rebuilt. Likewise, do not treat a fresh deployment timestamp as proof that the article's underlying information was reviewed. In an audit report, list the source of each date and the event it represents.

For example, an article corrected on Monday may be deployed on Tuesday and fetched on Wednesday. Those are three legitimate times. Labeling them all “last updated” loses useful evidence. A reviewer should be able to reconstruct when the correction was authored, when it became public, and when it was observed.

The sitemap guide covers URL discovery separately. This separation keeps delivery engineering focused on serving current content and editorial work focused on maintaining accurate information.

Frequently asked questions

Is a 304 response a crawler error?

Not by itself. In a conditional request, it can indicate that a client should reuse its stored representation. Review the original request and stored body before interpreting it.

Must every page have an ETag?

This audit does not require a particular validator. The operational requirement is that content changes reach clients correctly, with a caching strategy appropriate to the site.

Can I copy lastmod into Last-Modified?

Do not assume the fields describe the same event. Verify how the page is generated and which changes each timestamp tracks before connecting them in code.

Does fixing validators make an AI answer update immediately?

No. It corrects your delivery behavior for clients that use it. Retrieval schedules and answer generation remain outside the scope of this test.

Use the free page check for the public page, and retain the baseline and post-edit responses when investigating freshness problems.

Filed under Access and robots.txt. Topics: caching, http-status, curl