Prerendering for crawlers: when dynamic rendering still helps

Prerendering for crawlers means your server spots a bot by its user agent and hands it a fully rendered HTML snapshot of a JavaScript page, while human visitors get the normal client-side app. Google calls this dynamic rendering and says it is a workaround, not a recommended solution. It still has a place: as a short-term bridge for a client-rendered site that cannot move to server-side rendering yet, provided the snapshots stay fresh, match what users see and are served to AI crawlers as well as Googlebot.

What a prerender service actually does

A client-side rendered page often arrives as an almost empty HTML shell plus a JavaScript bundle. A browser runs the bundle and builds the content. A crawler that does not run JavaScript sees only the shell, which is the empty HTML problem.

A prerender setup puts a step in front of that:

1. A middleware, CDN worker or server rule reads the User-Agent header of each request.
2. If it matches a list of known crawlers, the request goes to a rendering service instead of your app.
3. The rendering service loads the page in a headless browser, waits for the JavaScript to finish, and saves the resulting HTML.
4. The crawler receives that HTML, usually from a cache, so later requests do not wait for a fresh render.
5. Everyone else gets the normal JavaScript app.

Hosted services such as Prerender.io work this way, as did Google's own open-source Rendertron, which its repository now marks as deprecated and no longer maintained. The prerender-node middleware README describes the same flow: check the user agent against a default crawler list, skip static resources such as .js and .css, then fetch the prerendered HTML.

The comparison of the last three in depth is in SSR, SSG and CSR compared for AI search.

What Google says about dynamic rendering

Google's page on dynamic rendering is now titled as a workaround. Three points from it matter:

• It is not the recommended fix. Google says dynamic rendering is "a workaround and not a recommended solution" because of the added complexity and resource requirements, and recommends server-side rendering, static rendering or hydration instead. Google's Search Central changelog records that on 6 February 2024 the page was updated to clarify that dynamic rendering is a deprecated workaround.
• Similar content is not cloaking. Googlebot generally does not treat dynamic rendering as cloaking "as long as your dynamic rendering produces similar content". Serving completely different content to users and crawlers can be considered cloaking; Google's example is a page about cats for users and a page about dogs for crawlers.
• Googlebot can render JavaScript itself. Google's JavaScript SEO basics explains that pages are queued for rendering in an evergreen headless Chromium, which may take a few seconds or longer. The same page says server-side or pre-rendering is still a great idea, because it makes a site faster for users and crawlers, and not all bots can run JavaScript.

So for Google Search alone, prerendering is mostly a speed and reliability aid. The stronger case for it today comes from other crawlers.

Why prerendering matters more for AI crawlers

Most AI crawlers did not run JavaScript when they were last measured. In Vercel and MERJ's December 2024 analysis, The rise of the AI crawler, none of the major AI crawlers rendered JavaScript, including OpenAI's crawlers, ClaudeBot and PerplexityBot. ChatGPT's and Claude's crawlers downloaded JavaScript files but did not execute them. That is a snapshot from one network at one time, and the JavaScript rendering guide covers what is and is not known since.

The practical result: on a client-rendered site, Googlebot may eventually see your content while a crawler that does not run JavaScript reads only the shell. A prerender layer closes that gap, but only for the bots it recognises.

The risks of prerendering for crawlers

Stale snapshots

Prerender services cache snapshots and re-render them on a schedule. Prerender.io, for example, calls this cache expiration: it sets how long a rendered page stays cached before it is re-rendered, so a one-day expiration re-caches each URL every 24 hours. Between re-renders, crawlers get the old version. On a shop, that can mean an old price or a product shown as in stock after it sold out, and an AI answer built from the snapshot repeats the old fact. Shorten the expiry for pages that change often, and trigger a re-render when content changes if your service supports it.

Content that differs from what users see

Differences creep in without anyone choosing them: a snapshot taken before an API call finished, a cookie banner or region selector captured over the content, a logged-out variant, or an A/B test arm. Small differences are normal. Missing main text, different prices or a different page are the "completely different content" Google warns about, and they also feed AI systems facts your visitors never see.

Wrong status codes

If your app shows a "not found" view but the server returns 200, the snapshot is cached as a normal page: a soft 404 served to every crawler. Prerender.io's docs on 404 pages cached as 200 recommend either returning a real 404 status from the server or adding a <meta name="prerender-status-code" content="404"> tag so the prerender layer passes the right status to crawlers. Redirects need the same care: the crawler should get the same status code a browser gets.

A bot list that stops at Googlebot

A bot list written for search engines and social preview bots will not include AI crawlers that appeared later. In prerender-node, matching is a case-insensitive substring test on the user agent, so a crawler whose name is not on the list falls through to the empty shell.

At the time of writing, the default list in the latest npm release of prerender-node (3.8.3) matches GPTBot, ChatGPT-User and anything containing Perplexity, but not OAI-SearchBot, ClaudeBot, Claude-SearchBot or Claude-User: its Anthropic entries are Anthropic/Claude and anthropic-ai, which none of those user agents contain. The unreleased master branch adds oai-searchbot and claudebot, but still not Claude-SearchBot or Claude-User. Those are the two agents that Anthropic's crawler article says may reduce a site's visibility in user search results if they are disabled. Hand-written nginx rules and CDN workers may have none of the AI crawlers. Check yours rather than assuming.

When prerendering is a sensible bridge

Prerendering makes sense when all of these are true:

• The site is client-side rendered and its important pages are public.
• Moving to server-side rendering or static generation is a real project that will take months.
• You can monitor the snapshots: freshness, status codes and parity with the live page.
• The prerender layer can be configured with the full list of crawlers you care about.

It makes less sense when the framework already supports server rendering (Next.js, Nuxt, SvelteKit, Astro and similar), when content changes by the minute, or when the pages sit behind a login anyway. In those cases the effort is better spent on rendering the HTML properly for everyone.

Treat prerendering as temporary. Write down which templates rely on it, and remove it template by template as each one moves to server rendering.

How to check that AI crawlers get the snapshot

The test is simple: request the same URL with different user agents and compare. Pick a sentence that only appears after JavaScript runs, such as a line from the main content or a product price.

URL="https://example.com/pricing"
PHRASE="Billed annually"

for UA in \
  "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
  "ClaudeBot" \
  "Claude-SearchBot" \
  "Claude-User" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)" \
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
do
  body=$(curl -s -A "$UA" "$URL")
  code=$(curl -s -o /dev/null -w "%{http_code}" -A "$UA" "$URL")
  bytes=$(printf '%s' "$body" | wc -c | tr -d ' ')
  if printf '%s' "$body" | grep -qF "$PHRASE"; then found=yes; else found=NO; fi
  printf '%s  %8s bytes  phrase=%-3s  %s\n' "$code" "$bytes" "$found" "${UA:0:60}"
done

The user agents for OpenAI, Perplexity and Googlebot are the documented strings. Anthropic's support article does not publish full strings for its bots, so the test sends the bare token, which is what substring matching looks for. The last line is a normal browser: it should show the small shell without the phrase, and every crawler line should show the phrase and a larger response. Any crawler line that looks like the browser line is not on your bot list. The curl guide has more variations.

Then run three more checks:

1. Status codes. Request a URL that does not exist with a crawler user agent. It should return 404, not 200.
2. Parity. Compare the title, H1, canonical, prices and main text in the crawler response with what a browser shows after loading. They should match.
3. Real traffic. A test from your laptop only proves the rule matches the header. If your firewall challenges spoofed bot names, you may see a challenge page instead, which is correct behaviour. Confirm with real requests in your logs, and check that those requests come from the operators' published IP ranges, as described in how to verify a real AI crawler.

How to add AI crawlers to the bot list

Where the list lives depends on the setup. In prerender-node 3.8.3, use the addUserAgents helper; on versions without it, push the tokens onto the exported prerender.crawlerUserAgents array:

const prerender = require("prerender-node");

prerender
  .set("prerenderToken", process.env.PRERENDER_TOKEN)
  .addUserAgents(["OAI-SearchBot", "ClaudeBot", "Claude-SearchBot", "Claude-User"]);

app.use(prerender);

For a self-hosted renderer behind nginx, the usual pattern is a map on the user agent that your routing then checks:

map $http_user_agent $prerender_ua {
    default 0;
    "~*googlebot|bingbot|applebot"              1;
    "~*gptbot|oai-searchbot|chatgpt-user"       1;
    "~*claudebot|claude-searchbot|claude-user"  1;
    "~*perplexitybot|perplexity-user"           1;
}

Match on the token (GPTBot, OAI-SearchBot), never on a full string with a version number, because operators change versions. OpenAI's crawler docs note that the version in its strings may change. Keep the list next to the AI crawler user agents list and review it when operators add agents.

One more detail: if a CDN caches pages in front of the prerender layer, make sure the cache key separates bot and browser responses. Otherwise a cached shell can be served to crawlers, or a cached snapshot to people.

Frequently asked questions

Is dynamic rendering cloaking?

Not in Google's view, as long as crawlers and users get similar content. Serving crawlers completely different content, such as a different topic or different prices, can be considered cloaking.

Does Google still support dynamic rendering?

Google still documents how it works and does not treat it as cloaking when the content is similar, but calls it a workaround and not a recommended solution. It recommends server-side rendering, static rendering or hydration for new work.

Do I need prerendering if Googlebot renders JavaScript?

For Google Search alone, often not, though rendering can be delayed. For AI crawlers it matters more, because the major ones did not run JavaScript in Vercel's December 2024 measurement and read only the raw HTML.

Is prerendering enough for AI search?

It removes one barrier: empty HTML. Crawlers still need to be allowed in robots.txt, get past your firewall, and find accurate, well-structured content. None of that guarantees an assistant will cite the page.

How often should snapshots be refreshed?

As often as the content changes in ways that matter. Pricing, stock and event pages need short expiry or a re-render on publish; evergreen articles can wait longer.

Check what crawlers receive today

Run the curl loop above on your three most important templates before deciding anything. If you want it checked for you, the free check tests a single page with no account, and a full audit runs deterministic checks across up to 500 pages and delivers a PDF with evidence and prioritized fixes.