Skip to content

Netlify block AI crawlers: User Agent Blocker limits

How to block AI crawlers on Netlify: what the User Agent Blocker extension does, its one-project limit, what it cannot catch, and how to pair it with robots.txt.

By Crawlito · 9 min read

In short

  • Netlify's User Agent Blocker extension uses an edge function to block requests from a preset list of AI and SEO crawlers, and Netlify says you can activate it for only one project at a time.
  • Netlify's docs say the extension's AI crawler list does not include every AI crawler, and its developer guide warns that some bots misreport their user agent, so a user-agent rule only stops bots that identify themselves.
  • Netlify's developer guide recommends using robots.txt and an edge function together: robots.txt for crawlers that honor it, a user-agent block for those that do not.
  • Blocking everything labeled AI also blocks search crawlers and user-triggered fetchers: OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers.
  • A curl request with a bot's user agent shows you what a user-agent rule returns, and it should show 200 for the search bots you want to keep.
On this page

On Netlify you can block AI crawlers in three ways: the User Agent Blocker extension, a robots.txt file, or your own edge function. The extension and the edge function refuse requests whose User-Agent header matches a list. robots.txt only asks. None of them can prove who is really calling, and a copied block list can cut off AI search bots you wanted to keep, so pick the entries on purpose.

This guide covers what the extension does and does not document, where a user-agent match falls short, and a safe default that blocks training crawlers and leaves search bots alone. It describes technical access only. Whether any assistant cites your site depends on much more than this.

What are the options for blocking AI crawlers on Netlify?

OptionEnforced?Who controls the listWhat Netlify documents
User Agent Blocker extensionYes, for matching user agentsA preset list you choose fromRuns on a Netlify Edge Function; one project at a time
robots.txt fileNo, it is a requestYouNetlify's guide says not all AI products respect it
Custom edge functionYes, for matching user agentsYouExample in Netlify's developer guide returns a 401

Netlify's guide recommends combining robots.txt with the edge function, so well-behaved crawlers get the polite signal and the rest get refused. The extension is the no-code version of the edge function half.

How does the User Agent Blocker extension work?

Netlify built the extension to block web requests from a preset list of common AI crawlers and SEO or search crawlers that you choose from. It uses a Netlify Edge Function to do the blocking, and Netlify frames the goal as better control over web request traffic to your site.

The text of Netlify's docs page leaves several things unsaid at the time of writing:

  • The response. The page does not say which status code a blocked request gets.
  • The exact list. The text does not name the user agents behind each option. A screenshot on the page shows some option names, such as GPTBot, GoogleOther and ClaudeBot, but it is a screenshot, not a list you can rely on.
  • Plans. It does not say which plans include the extension.
  • robots.txt. The text does not mention the file, although the same screenshot shows a setting about adding the user agents to the site's robots.txt next to one for blocking requests with an edge function. The docs do not explain how that setting behaves.

Treat those as things to test on your own site, not as defaults you can assume. After you deploy, open your live /robots.txt and check what it contains. Netlify also states that the AI crawler list does not include all possible AI crawlers or bots.

How do you set it up, and what is the one-project limit?

These are the steps from Netlify's docs:

  1. In your dashboard, open Extensions in the left sidebar.
  2. Search for User Agent Blocker and select it.
  3. Select Install. This makes it available to all projects on your Netlify team.
  4. Open the project where you want it active.
  5. Under Extensions in the project's left navigation, select User Agent Blocker.
  6. Under Block User Agents, select the traffic to block. Netlify says to select all of the AI crawler options to target AI crawlers and bots.
  7. Select Redeploy site and confirm with Save.
  8. Select Enable analytics and deploy site to activate it.

Once the deploy succeeds, Netlify says blocking starts immediately.

The limit: you can activate the extension for only one project at a time. Installing it for the team does not change that. If you run several Netlify sites, the extension covers one of them. For the others, use an edge function in each repository, as in the next sections. Netlify's guide also suggests packaging the function as a Netlify Integration when you manage multiple sites.

Before you select every AI crawler option, read what each one covers in your dashboard. Since the docs text does not list the user agents, run the curl test below after the deploy and confirm that the search bots you want still get a 200.

What can a user-agent match not stop?

A user-agent rule reads one request header, and the sender chooses that header. That has four consequences.

  • Bots that misreport their name. Netlify's guide says some bots misreport their user agent, so filtering on it alone is not enough.
  • Bots that are not on the list. The extension's list is partial by Netlify's own account, and new bots appear often.
  • AI use with no separate user agent. Google's Google-Extended is a standalone robots.txt token. Google says it has no separate HTTP user agent string, so no edge rule can see it. Netlify's user agent categories page makes a related point: some vendors, notably Google, do not use a separate user agent when crawling for AI purposes.
  • Fakes in both directions. Anyone can send GPTBot in a header. A match tells you a request claims to be GPTBot, not that it is. To tell real from fake, check the source IP against the operator's published list, as in how to verify GPTBot IPs and spot fake AI crawlers.

There is also a limit that is not about spoofing. RFC 9309 says robots.txt rules are not a form of access authorization and are not a substitute for content security. So robots.txt on its own cannot enforce anything, and a user-agent block cannot see who is behind the header. If a page must be private, put it behind real authentication.

How do you combine robots.txt with edge-level blocking?

Use robots.txt to state your policy and the edge function to enforce it for the training crawlers you have named. Here is a robots.txt that blocks training and leaves search alone:

# Training: opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Put it in the root of the published site so it serves at /robots.txt. A crawler that has a named group follows that group and ignores the * group, as robots.txt for AI crawlers explains.

Then add an edge function. Netlify's guide example is the starting point, with a few changes: a short list of training tokens, a safe default when the User-Agent header is missing, a 403 instead of a 401, and /robots.txt left outside the rule. Save it as netlify/edge-functions/block-training-bots.ts:

ts
import type { Config } from "@netlify/edge-functions";

// Training crawlers only. Match the token, not the full string.
const BLOCKED = ["gptbot", "claudebot", "ccbot"];

export default async (request: Request) => {
  const ua = (request.headers.get("user-agent") ?? "").toLowerCase();

  if (BLOCKED.some((token) => ua.includes(token))) {
    return new Response("Forbidden", { status: 403 });
  }
  // Returning nothing lets the request continue.
};

export const config: Config = {
  path: "/*",
  excludedPath: ["/robots.txt"],
};

Why those changes:

  • Header can be missing. The guide's example calls toLowerCase() on the header value directly. When a request has no User-Agent, headers.get returns null and that call throws. The ?? "" avoids it.
  • Status code. The guide returns 401 with an empty body. I used 403 because the request is refused, not asking for credentials. Both are 4xx responses.
  • robots.txt stays reachable. Netlify's request processing order runs edge functions before static files, so a function that matches /robots.txt serves its own response instead of your file. Under RFC 9309, a crawler that gets a 4xx for robots.txt may treat it as unavailable and access anything the server allows. Anthropic warns that blocking by IP address can stop its crawler from reading robots.txt, which weakens an opt-out. Excluding the file keeps your stated policy readable.

If you prefer a TOML declaration, Netlify's edge function declarations support path and excludedPath there too. The server-level version of this idea is in blocking AI crawlers in nginx or .htaccess.

Which AI bots should you not block?

Do not copy a long "AI bots" list without reading it. Netlify's example agents.json in its examples repository, as it stands at the time of writing, includes GPTBot, ClaudeBot, CCBot and Google-Extended, but also ChatGPT-User and PerplexityBot. It does not include OAI-SearchBot or Claude-SearchBot. The purposes differ:

AgentPurposeWhat blocking it does
GPTBotCrawls content that may train OpenAI modelsOpts out of training; OpenAI says the settings are independent
OAI-SearchBotSurfaces sites in ChatGPT searchOpenAI says opted-out sites are not shown in ChatGPT search answers
ChatGPT-UserVisits a page when a user's request needs itOpenAI says robots.txt rules may not apply; it is not used to decide search inclusion
ClaudeBotCollects content that could contribute to trainingOpts out of training
Claude-SearchBotImproves search result qualityAnthropic says blocking it may reduce visibility in search results
PerplexityBotSurfaces and links sites in Perplexity searchPerplexity says it is not used to crawl content for AI foundation models
Perplexity-UserFetches a page when a user asksPerplexity says it generally ignores robots.txt

So the safe default is the one above: refuse GPTBot, ClaudeBot and CCBot, and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Decide the user-triggered agents separately. Netlify's categories page says traffic in its ai-agent category is usually low volume and initiated by a person, and that it is generally not advisable to block it. That is a choice for you, not a rule, and the trade-off is that a person who asks an assistant about your page gets nothing. The full policy is in how to allow AI search but block AI training, and every token is in the AI crawler user agents list.

Also avoid blocking by Netlify's crawler;ai category as a shortcut. It is described as high-volume AI provider crawlers from vendors such as OpenAI, Anthropic and Perplexity, and the page does not separate training bots from search bots.

How do you test the block with curl?

Send a request with the bot's user agent and read the status code. A header you send yourself is all a user-agent rule looks at, so this test is faithful for the rules above. Netlify says that on deployed sites it overrides any Netlify-Agent-Category header you send, so test by setting the User-Agent, not that header.

bash
for ua in "GPTBot/1.4" "ClaudeBot/1.0" "OAI-SearchBot/1.4" "Claude-SearchBot/1.0" "PerplexityBot/1.0"; do
  printf "%s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://example.com/
done

With the edge function above you should see 403 for GPTBot and ClaudeBot and 200 for the search bots. Then confirm the policy file is still served to a blocked bot:

bash
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.4" https://example.com/robots.txt

That should return 200. If you use the extension instead, run the same loop to learn what it returns, since the docs do not say. A test from your own IP does not tell you how a CDN or firewall rule that depends on IP reputation treats the real crawler. See your page as an AI crawler with curl covers more of this method.

Frequently asked questions

It can, depending on which options you select. OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. Netlify's docs text does not list which user agents each option covers, so test OAI-SearchBot's user agent after you deploy.

Can I use the extension on more than one Netlify site?

Not at the same time. Netlify says you can activate it for only one project at a time. For other projects, add an edge function like the one above to each site.

Is robots.txt enough on Netlify?

For crawlers that honor it, yes. Netlify's guide says not all AI products respect it, and RFC 9309 says robots.txt is not access authorization. Add an edge rule for crawlers you must refuse, and keep in mind that spoofed user agents get past both.

What status code does the extension return?

Netlify's docs page does not say. The example edge function in Netlify's developer guide returns 401 with an empty body. Check your own site with curl.

How long until a change takes effect?

Netlify says extension blocking starts immediately once the deploy succeeds. For robots.txt, OpenAI says that for OAI-SearchBot it can take about 24 hours for its systems to adjust after an update.

Check what your site returns

After you deploy, the free check reads one page with no account and shows the robots.txt rules for each named AI crawler, so you can confirm that the search bots you meant to allow are not disallowed. It runs from Crawlito's servers, not from any crawler's IP ranges. For a whole site, an audit covers up to 500 pages.

Filed under Platform guides. Topics: netlify, bot-protection, ai-crawlers, robots-txt