Netlify Prerender extension: which crawlers get rendered HTML
How the Netlify Prerender extension picks which requests get rendered HTML, how to test each user agent category, and how to move off legacy prerendering.
By Crawlito · 9 min read
In short
- The Netlify Prerender extension replaced the legacy built-in prerendering feature, which Netlify scheduled to be disabled plan by plan between January 20 and March 17, 2026.
- The extension adds an edge function that runs only for relevant user agents requesting HTML and rewrites those requests to a function that renders the page in headless Chromium.
- Netlify says the edge function decides using its user agent categorization header, Netlify-Agent-Category, which sorts requests into categories such as crawler, ai-agent, page-preview, tooling and browser.
- Netlify overrides any Netlify-Agent-Category header a client sends on a deployed site, so you test a category by changing the User-Agent header, not by setting the category yourself.
- The extension needs Node.js 20 or later, is free to install, and bills its function invocations and traffic like any other function on your plan.
On this page
- What replaced Netlify's legacy prerendering
- How the extension decides who gets prerendered HTML
- How to test a user agent category on your site
- What to do when a bot is not categorised the way you need
- How to migrate from legacy prerendering
- Frequently asked questions
- Check what crawlers receive from your Netlify site
The Netlify Prerender extension serves rendered HTML to crawlers, AI agents and link preview services, while people get your normal JavaScript app. Netlify says it picks those requests using its own user agent categorization header, not a list you maintain. A request whose User-Agent lands in a category the extension targets gets a page rendered in headless Chromium; everything else gets your normal response, which on a client-rendered app is the near-empty shell. Netlify does not publish the exact categories the extension matches, so the job is to check which categories your important crawlers land in, and test the result with curl.
This post covers the Netlify specifics. The general case for and against serving bots a snapshot is in prerendering for crawlers. Blocking bots on Netlify is a separate topic, covered in Netlify block AI crawlers.
What replaced Netlify's legacy prerendering
For years Netlify had a built-in prerendering toggle. On 16 December 2025 Netlify's changelog announced the Prerender extension as generally available and deprecated the legacy feature on this timeline:
| Date | Legacy prerendering disabled for |
|---|---|
| January 20, 2026 | Free plans (legacy or credit-based) and the legacy Starter plan |
| February 17, 2026 | Personal plan and Pro plans (legacy or credit-based) |
| March 17, 2026 | Enterprise and all other specialized plans |
All three dates have passed. If a project relied on the legacy toggle and nobody installed a replacement, crawlers that do not run JavaScript should now be getting the unrendered HTML shell, which is the empty HTML problem.
Netlify's announcement post gives the reason for the rebuild: the old feature was a black box. When its headless browser failed or ran slowly, you had no logs and no settings. The new extension is built from ordinary platform parts you can see:
- An edge function that runs only for relevant requests, meaning a relevant user agent asking for HTML, and rewrites them to the prerender function.
- A serverless function that loads the page in a headless browser and returns the rendered HTML with caching headers.
- A Netlify Blob that stores the extension's optional settings, so changes other than enabling or disabling it do not need a redeploy.
Because both functions run in your account, their logs and metrics appear in Netlify's observability pages like any other function. Netlify says the extension is free on all plans, and the function invocations and traffic it uses are billed normally.
There is also a third-party option. The Prerendering docs describe a Prerender.io extension that needs a Prerender.io API key. The docs page still mentions a Pro or Enterprise plan for it, while a January 2026 changelog entry says it is now available to all Netlify customers. Check your plan page if you choose it.
How the extension decides who gets prerendered HTML
Netlify's announcement says the edge function intercepts relevant requests "based on the new user-agent categorization header". That header is Netlify-Agent-Category. Netlify sets it on requests to deployed projects, mostly by matching the raw User-Agent against known substrings. The value is a category, optionally followed by a sub-category:
<category>[;<optional sub-category>]The user agent categories page defines these categories at the time of writing:
| Category | Sub-categories | What lands here, per Netlify |
|---|---|---|
crawler | general, ai, social, seo | Bulk crawlers. ai is high-volume AI provider crawlers such as OpenAI, Anthropic and Perplexity; GPTBot/ is matched here |
ai-agent | user, search | AI services fetching pages to answer a prompt, matched by substrings such as ChatGPT-User/ and Claude-User/ |
page-preview | general | Link preview fetchers such as Slackbot, WhatsApp, Discord and Google Web Preview |
tooling | monitoring, netlify-service, prerender, custom-code, general | Scripts, monitors, HTTP libraries such as curl, and unmapped agents containing bot or api |
browser | none | Agents that parse as a known browser |
other | none | Provided but not recognised |
none | none | No User-Agent header at all |
Netlify's docs say the extension serves rendered HTML to "crawlers, agents, and preview services". Those three groups line up with the crawler, ai-agent and page-preview categories. The public docs do not publish the exact category match the extension's edge function uses. Its full settings are on the extension details page inside the Netlify app. So treat the mapping as something to confirm on your own site, which the next section shows how to do.
Two details from the categories page matter for AI search:
- The page does not say where AI search crawlers land.
crawler;aiis described only as high-volume AI provider crawlers, andai-agent;searchas an agent doing retrieval for a built-in search feature. The page does not say whether OAI-SearchBot, Claude-SearchBot or PerplexityBot fall into either, so test each one. - Google has no separate AI user agent. Netlify notes that Google does not use a separate user agent for AI crawling. Googlebot is the agent that matters for AI Overviews and AI Mode, and it renders JavaScript itself, as described in the React SPA guide.
How to test a user agent category on your site
You cannot test by sending the category header yourself. Netlify's docs say that on a deployed project it overrides any Netlify-Agent-Category header a client sends. Set the User-Agent instead. Locally, with netlify dev or the Vite plugin, the header is neither set nor overridden.
Step 1: see which category Netlify assigns
The quickest way to see the category is a tiny edge function on a path nobody else uses. It echoes the header back. Save it as netlify/edge-functions/agent-category.ts:
import type { Config } from "@netlify/edge-functions";
// Debug only: returns the category Netlify assigned to this request.
export default async (req: Request) => {
const category = req.headers.get("netlify-agent-category") ?? "not set";
return new Response(category + "\n", {
headers: { "content-type": "text/plain", "cache-control": "no-store" },
});
};
export const config: Config = { path: "/_debug/agent-category" };Deploy it to a preview, then loop over the agents you care about. One caveat: Netlify's edge function processing order runs edge functions generated by enabled integrations before your own inline ones, so if the extension is already on, it may intercept some agents before this function runs. If a line comes back as HTML instead of a category, run the check on a deploy made before you enabled the extension.
SITE="https://deploy-preview-12--example.netlify.app"
for UA in \
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
"ClaudeBot/1.0" "Claude-SearchBot/1.0" "Claude-User/1.0" \
"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
"Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)" \
"Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)" \
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
do
printf '%-28s %s' "${UA:0:28}" "$(curl -s -A "$UA" "$SITE/_debug/agent-category")"
echo
doneOpenAI and Perplexity publish the example strings used above, and note that version numbers may change. Anthropic's crawler article names its agents but does not publish full strings, so the loop sends a token with a version. Netlify's example substring for Claude is Claude-User/ with the slash, so a bare Claude-User may not match the same way. Check your logs for the exact strings the real crawlers send. Remove the debug function when you are done.
Step 2: confirm the rendered HTML
Knowing the category is half the test. The other half is whether the response contains content that only appears after JavaScript runs. Pick a phrase from the main content and compare:
URL="https://example.com/pricing"
PHRASE="Billed annually"
for UA in "GPTBot/1.4" "OAI-SearchBot/1.4" "ChatGPT-User/1.0" "Slackbot-LinkExpanding 1.0" \
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"; do
body=$(curl -s -A "$UA" "$URL")
if printf '%s' "$body" | grep -qF "$PHRASE"; then found=yes; else found=NO; fi
printf '%-30s %7s bytes phrase=%s\n' "${UA:0:30}" "$(printf '%s' "$body" | wc -c | tr -d ' ')" "$found"
doneThe browser line should show the small shell without the phrase. Every agent you want served should show the phrase. If an agent's line looks like the browser line, either its category is not one the extension prerenders, or the render failed. The prerender function's logs in Netlify should help you tell which. A first request can be slower than later ones, because the function loads the page in a headless browser when it is requested.
What to do when a bot is not categorised the way you need
Netlify's categories page describes what happens to unknown agents. An unrecognised User-Agent containing bot or api goes to tooling;general. Anything else that is not a known browser goes to other. Netlify says the share of requests in other should be very low, under 1 to 2%. An agent in either bucket probably does not get prerendered HTML, since neither is a crawler, agent or preview category.
The public docs do not describe a way to add your own agents to the extension's match. Your options, in order of how much they fix:
- Check the extension's settings. Open the Prerender extension in your project's sidebar and read what it lets you configure. The details page is where Netlify points for advanced options, and those settings are not reproduced in the public docs.
- Tell Netlify. Netlify's announcement says the extension comes with community support through its forums. Post the exact
User-Agentstring from your logs and the category it received. - Remove the need for prerendering on the pages that matter. Build-time prerendering or server rendering gives every client the content, whatever its category. The comparison is in SSR, SSG and CSR for AI search.
Keep the content the same for every category while you do this. Serving crawlers a rendered copy of the page users see is fine. Serving them different text is the risk covered in serving AI crawlers different content.
How to migrate from legacy prerendering
These steps come from Netlify's changelog, announcement post and docs:
- Check whether you need it. Netlify offers a checker at
do-you-need-prerender.netlify.app. If your framework already server-renders or statically generates pages, you may not need the extension at all. - Turn off the legacy feature. Go to Project configuration > Build & deploy > Post processing > Prerendering and disable it if it is still on.
- Set Node.js 20 or later. The docs say older versions can make the prerender function fail, for example with Chromium launch errors. Set it with a
.node-versionfile, an.nvmrcfile or theNODE_VERSIONenvironment variable. - Install the extension. A Team Owner installs it from the extensions library at
app.netlify.com/extensions/prerender. Installing it makes it available to all projects without turning it on for any of them. - Enable it per project. Open the extension in the project's sidebar, select Enable prerendering, save, and redeploy.
- Test. Run both curl loops above on your most important templates and a URL that does not exist.
If you used the legacy feature as a proxy to an external prerendering vendor, Netlify's changelog asks you to contact its support about your options. If that vendor was Prerender.io, the January 2026 changelog says you need to move to the separate Prerender.io extension.
Netlify's post adds one point on caching: the extension works with Netlify's fine-grained cache. If the original content is purged, by a new deploy or an on-demand invalidation, the prerendered version is purged too. That means snapshot freshness follows your deploys and purges, which reduces the stale snapshot risk common to prerendering setups.
Frequently asked questions
Does the Netlify Prerender extension cost money?
Netlify says the extension is free on all account plans. The function invocations and traffic it uses are billed normally under your plan.
Can I set Netlify-Agent-Category to test the extension?
No. On a deployed site Netlify overrides any value a client sends, so change the User-Agent header instead. Locally, with netlify dev, the header is not set and you can set it yourself.
Does GPTBot get prerendered HTML?
Netlify's categories page puts GPTBot/ in the crawler category, and the extension targets crawlers. The public docs do not publish the extension's exact match, so confirm it with the curl test on your own site.
Is my legacy prerendering still working?
Not if it was the built-in Netlify feature. Netlify's timeline scheduled it to be disabled for all plans by March 17, 2026, so you need the extension, Prerender.io or server rendering instead.
Do I need prerendering if I use Next.js or Astro on Netlify?
Usually not, if your pages are server-rendered or statically generated, because the content is already in the HTML. Check a page's raw HTML for your main text before adding a prerender layer.
Check what crawlers receive from your Netlify site
After enabling the extension, run the free check on one important page. It is a single-page check and needs no account. For a whole Netlify site with many templates, an audit covers up to 500 pages and gives you a PDF with evidence and prioritized fixes.
Filed under Platform guides. Topics: netlify, rendering, javascript, ai-crawlers