Skip to content

CDN caching and AI search: diagnosing stale page content

Find out whether your CDN is serving outdated HTML to crawlers. Compare public responses, trace cache layers, and verify fixes after publishing.

By Crawlito · 6 min read

In short

  • A successful deployment does not prove that every public cache serves the new HTML.
  • Compare a specific changed sentence in public responses before blaming a search engine for stale information.
  • Purge the affected representation and fix invalidation so the next content update reaches visitors too.
On this page

CDN caching can leave crawlers reading yesterday's HTML even when your CMS and application show the latest version. To diagnose it, compare a specific changed fact in the public response with the expected published content, then identify which cache still holds the old representation. Changing a sitemap date cannot repair stale HTML delivered by your own infrastructure.

This guide is for a particular symptom: the same public URL sometimes returns an old price, policy, paragraph, or robots instruction. It does not assume that an assistant quoting old information fetched your site recently. First establish what your server actually delivers; investigate search freshness separately afterward.

Build a reproducible freshness check

Choose a short, distinctive sentence that changed in a recent release. A sentence is a better diagnostic marker than a screenshot of the whole page. For example, in a hypothetical shipping policy update, the old sentence says orders ship in five working days and the new sentence says three working days. Record the publication time and the exact public URL.

Save a public response without signing in:

bash
curl --silent --show-error --location \
  --dump-header public-headers.txt \
  --output public-page.html \
  'https://www.example.com/shipping'

Open the saved file and search for both versions of the sentence. Also inspect the response headers and final URL. A login redirect, regional storefront, or preview route is a different explanation from a stale cache. Keep those cases separate in the incident notes.

Repeat the same request once after the expected cache lifetime, if that lifetime is known. Do not continuously request the URL or append random query strings in a loop. A cache-busting parameter can select a different cache entry and hide the precise problem experienced on the canonical URL.

Our raw versus rendered HTML guide helps distinguish cached server content from browser-side updates. If the saved HTML is old but JavaScript replaces it with new text, both a cache problem and a rendering dependency may exist.

Map the caches between the editor and visitor

Draw the actual publishing path. Many teams have more than one cache: a CMS response cache, application data cache, generated page, reverse proxy, CDN, and browser cache. An edge purge only affects the layer it reaches. If the application still generates old HTML, the CDN can immediately store that old HTML again.

Use a small investigation table:

ObservationQuestion to investigateEvidence to save
CMS has new text, application preview does notDid publication invalidate application data?CMS revision and preview output
Application output is new, public response is oldIs a shared cache retaining the page?Public headers and body
One region is old and another is newAre cache locations updating consistently?Timestamp and location of each test
Only signed-in visitors see the updateIs the anonymous representation stale?Cookie-free request and authenticated comparison

These are investigation paths, not conclusions based on one header. Provider-specific cache status fields can be helpful, but their names and meanings vary. Read the documentation for your actual CDN before interpreting a hit, miss, bypass, or stale label.

RFC 9111 defines HTTP caching and response freshness. It distinguishes whether stored content is fresh from whether it can be reused under particular conditions. Application caches can add their own behavior on top of those HTTP rules, which is why an infrastructure diagram is useful.

Inspect the content and its indexing controls together

When a page is stale, do not compare only its main paragraph. Save the title, canonical link, robots meta tag, and any X-Robots-Tag response header too. A deployment may have corrected an accidental noindex while the cached response still carries it. A page can therefore look repaired in the editor and remain restricted at the public edge.

Similarly, a product page may have a current visible price but stale structured data if separate templates or caches generate those fields. Compare the fields from one response captured at one time. Mixing a screenshot from today with JSON-LD copied yesterday creates a mismatch that may not exist in either real response.

Review staging noindex leaks when old deployment settings appear. For cached challenge pages, use the WAF troubleshooting guide: a challenge cached as a successful page needs a different fix from an outdated article.

Never share a raw response containing account information, tokens, or private application state in a public issue. A useful incident attachment contains the relevant public sentence and headers with unnecessary identifiers removed.

Fix the publishing path, then purge precisely

Start at the first layer that still produces the old content. If the CMS webhook failed, repair or retry that publication event. If a build used an older revision, generate the correct revision. If the public edge alone is stale, invalidate the affected page and any deliberately linked representations, such as a cached category listing.

Avoid treating a whole-site purge as the routine publishing workflow. It can conceal missing dependencies and cause unnecessary origin load. Define which artifacts a change should update. A price edit may affect a product page, its category summary, and a feed. An article correction may affect only the article and the feed description.

For every recurring content type, write an invalidation contract in plain language:

  1. Identify the event that begins publication.
  2. List the public artifacts that contain the changed information.
  3. Name the cache or generator responsible for each artifact.
  4. Describe how completion is checked.
  5. Assign an owner when the expected update does not appear.

This contract is useful even without a sophisticated deployment system. It converts “the editor saved successfully” into “the public artifact contains the intended revision.” Those are different acceptance criteria.

Verify the next edit as well as the current repair

After invalidation, fetch the original canonical URL again with the same conditions. Confirm the new sentence, expected metadata, and correct response body. Then verify a second, ordinary content update through the normal publishing workflow. A manual purge can solve today's incident while leaving the next publication broken.

Create a short release record with the content revision, publication time, expected sentence, first confirmed public response, and any remaining regional uncertainty. Do not label the release globally verified if you tested only one public route from one location. State the actual scope.

Keep search observations in a separate column. A fresh public response demonstrates delivery, while an updated search snippet demonstrates a different downstream event. An assistant's answer might also rely on another source. This distinction prevents repeated cache changes after your infrastructure is already working correctly.

For a small site, a single representative article and a commercial page are a reasonable starting sample. Add templates when they use different publication paths. The goal is coverage of different mechanisms, not a large collection of identical requests.

Frequently asked questions

Does clearing my browser cache fix what crawlers receive?

No. It changes your local browser state. A crawler reaches the public infrastructure independently, so verify the anonymous public response and the relevant shared caches.

Should I disable caching to keep content fresh?

Not as a default response. Fix publication and invalidation first, then choose cache lifetimes appropriate to how often each kind of content changes.

Does a fresh page mean an assistant has updated its answer?

No. It proves that the tested request receives the new page. Search processing, retrieval timing, and the assistant's other sources remain separate questions.

Can a sitemap timestamp clear a stale CDN entry?

No. A sitemap describes URLs and changes; it does not invalidate your delivery infrastructure. Repair the public response before using discovery signals to announce an update.

Start with the affected URL in the free page check, and keep the saved response alongside your publishing evidence.

Filed under CDN and bot protection. Topics: cdn, caching, http-status, ai-search