Skip to content

Block AI crawlers in nginx or .htaccess: user agent rules

Block AI crawlers in nginx or .htaccess by user agent: copy-paste rules for GPTBot, ClaudeBot and CCBot that leave search bots and robots.txt reachable.

By Crawlito · 9 min read

In short

  • To block AI crawlers at the server, match their user agent token and return 403: in nginx with a map on $http_user_agent, in Apache with a RewriteCond on %{HTTP_USER_AGENT} and a RewriteRule with the [F] flag.
  • Match only the crawlers you have opted out of, such as GPTBot, ClaudeBot and CCBot. A loose pattern such as Claude or bot also catches Claude-SearchBot, Googlebot and other crawlers you may want to keep.
  • Leave /robots.txt outside the block: RFC 9309 lets a crawler treat a 4xx response there as no restrictions, and Anthropic warns that blocking by IP address can keep its bots from reading robots.txt.
  • A user agent rule only stops bots that tell the truth. Anyone can send the string GPTBot, so verify the source IP against the operator's published list before you decide a request is real or fake.
  • A server rule is enforcement, robots.txt is a request: it can stop ChatGPT-User, where OpenAI says robots.txt rules may not apply, but it also stops people who ask ChatGPT about your page.
On this page

To block AI crawlers in nginx or Apache, match the crawler's name in the User-Agent header and refuse the request with a 403. In nginx that is a map on $http_user_agent plus a return 403. In Apache it is a RewriteCond on %{HTTP_USER_AGENT} followed by a RewriteRule with the [F] flag, usually in .htaccess. Match only the crawlers you have opted out of, keep /robots.txt reachable, and test with curl before you rely on it.

A server rule is blunt. It cannot tell a real GPTBot from a script that copies its name, and a pattern that is too loose will refuse crawlers that bring you visitors. The sections below show the rules and the checks that keep them narrow.

When is a server rule worth it, and when is robots.txt enough?

robots.txt is a request. RFC 9309 says its rules are not a form of access authorization, so a crawler can read your file and ignore it. For the documented crawlers from OpenAI and Anthropic, robots.txt is the opt-out mechanism the operators describe, and it is the right first step. Our robots.txt guide for AI crawlers covers the syntax, and allowing AI search while blocking AI training has a ready-made file.

A server rule makes sense when:

  • A bot you have disallowed keeps requesting pages, and you want the requests to stop costing you bandwidth or database work.
  • The operator says robots.txt may not apply to the agent. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person triggered the fetch.
  • You need the rule to hold for a specific directory, such as a members area, without relying on the bot's cooperation.

Keep both. The robots.txt Disallow is the signal the operators document. The server rule is enforcement behind it. The operator pages cited in this post describe robots.txt, not an HTTP 403, as the way to opt out, so do not remove the robots.txt rule once the server rule works.

Which user agent tokens should you match?

Match on a token, not the full string. Version numbers and surrounding text change: OpenAI's current examples show GPTBot/1.4 and OAI-SearchBot/1.4. Under RFC 9309 a crawler's product token should match a substring of its identification string, so a case-insensitive substring match on the token is the right shape.

TokenOperatorJobBlock at the server?
GPTBotOpenAITrainingReasonable, if you have opted out
ClaudeBotAnthropicTrainingReasonable, if you have opted out
CCBotCommon CrawlOpen datasetReasonable, if you have opted out
OAI-SearchBotOpenAIChatGPT searchNo: OpenAI says sites that opt out are not shown in ChatGPT search answers
Claude-SearchBotAnthropicClaude searchNo: Anthropic says blocking may reduce visibility in Claude's search results
PerplexityBotPerplexityPerplexity searchNo, if you want to be in its results
ChatGPT-User, Claude-User, Perplexity-UserOpenAI, Anthropic, PerplexityA person asked for your pageYour call: blocking stops those requests
Googlebot, bingbotGoogle, MicrosoftWeb searchNo

Three traps are worth knowing before you write a pattern.

Loose patterns catch the wrong bots. ClaudeBot does not appear inside Claude-SearchBot, but Claude does. A pattern of bot matches Googlebot and bingbot too. Write the full token.

Some tokens never appear in logs. Google says Google-Extended has no separate user agent string, and Apple says Applebot-Extended does not crawl webpages. They exist only as robots.txt tokens, so a server rule cannot match them. To opt out of Gemini or Apple model training, use robots.txt. Blocking Googlebot or Applebot at the server affects your presence in Google Search or in Apple's search features instead.

User-triggered agents need a decision. The three user-triggered agents (ChatGPT-User, Claude-User and Perplexity-User) visit because a person asked a question that needs your page. A server rule stops them even where robots.txt would not, and the cost is that the person gets nothing from your page. OpenAI says ChatGPT-User does not decide whether you appear in ChatGPT search, so the choice does not touch search eligibility. For the full list of tokens, see the AI crawler user agents list.

nginx: a map on the user agent that returns 403

The nginx map module creates a variable whose value depends on another variable. The map directive is valid in the http context only, so it goes outside any server block. A prefix of ~* makes a regular expression match case-insensitively, and the default value applies when nothing matches.

nginx
# In the http block (often a file in conf.d that nginx.conf includes there)
map $http_user_agent $ai_training_ua {
    default      0;
    ~*GPTBot     1;
    ~*ClaudeBot  1;
    ~*CCBot      1;
}

map $uri $is_robots_txt {
    default      0;
    /robots.txt  1;
}

# Block only when the user agent matches and the path is not /robots.txt
map "$ai_training_ua$is_robots_txt" $block_ai_training {
    default  0;
    "10"     1;
}

server {
    server_name example.com;
    # your existing listen, root, TLS and location settings stay as they are

    if ($block_ai_training) {
        return 403;
    }
}

Why it is built this way:

  • A map source can combine several variables into one string, so the third map reads 1 followed by 0 (a matching user agent on a path that is not robots.txt) and returns 1 only for that pair.
  • The map module documentation shows a map on $http_user_agent; acting on the result with if ... return 403 is the standard way to use the rewrite module. Here the if sits at the server level. The nginx rewrite module runs server-level directives first, before it picks a location, so the check covers every location on the server, including a regex location that hands requests to PHP.
  • nginx evaluates a map variable only when something uses it, so the maps cost nothing on a server where the if is absent.
  • To protect only part of the site, move the if into a narrower location, such as location /members/. Then it applies only when that location is chosen.

Run nginx -t to test the syntax, then reload nginx. To block a different bot, add one line to the first map.

Apache: mod_rewrite rules in .htaccess

On Apache, use mod_rewrite. The pieces are documented: RewriteCond can test %{HTTP_USER_AGENT}, the [NC] flag makes the test case-insensitive, conditions are combined with AND unless you add [OR], a - substitution leaves the path untouched, and the [F] flag returns a 403 Forbidden and stops processing.

apache
RewriteEngine On

# Crawlers you have opted out of (case-insensitive)
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot) [NC]
# Never block robots.txt
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule ^ - [F]

Both conditions must hold, because Apache combines them with AND by default. A request is refused only when the user agent matches and the path is not /robots.txt. Apache's documentation describes %{REQUEST_URI} as the path of the requested URI, and says to use it in a RewriteCond when you want to match the full URL path in a per-directory rule.

Three details decide whether this works on your host:

  1. .htaccess must be enabled. The default AllowOverride is None, which makes .htaccess files ignored. mod_rewrite directives in .htaccess need AllowOverride FileInfo, and the Options for the directory must include FollowSymLinks or SymLinksIfOwnerMatch. On shared hosting your provider usually sets this up.
  2. Prefer the main configuration if you can edit it. Apache's own .htaccess guide says to put configuration in the main server config when you have access, because .htaccess files are read on every request and slow the server. The same RewriteCond and RewriteRule lines work in a <VirtualHost> or <Directory> block. Rewrite settings are not inherited by virtual hosts, so each one needs its own RewriteEngine On.
  3. Order matters if a CMS writes its own rewrite block into .htaccess. Rules run in file order, so put these lines above it, so the 403 is decided before the request is routed to the CMS's front controller.

How do you test the rule with curl?

Test three things: the bots you blocked get a 403, the bots you kept get a normal answer, and robots.txt is open to everyone. The rules match on a substring, so short strings are enough for testing.

bash
site=https://example.com

for ua in "GPTBot/1.4" "ClaudeBot/1.0" "CCBot/2.0" \
          "OAI-SearchBot/1.4" "Claude-SearchBot" "PerplexityBot/1.0" \
          "Googlebot/2.1" "Mozilla/5.0 (Macintosh) Safari/605.1.15"; do
  printf '%-45s page %s  robots.txt %s\n' "$ua" \
    "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" "$site/")" \
    "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" "$site/robots.txt")"
done

You want 403 on the page and 200 on robots.txt for the first three lines, and 200 on both for every other line. If Claude-SearchBot or Googlebot gets a 403, a pattern is too loose.

Two checks catch most mistakes:

  • Run the browser line last, through the public hostname. If a cache or CDN in front of your server does not vary on User-Agent, a stored 403 could reach ordinary visitors. A normal 200 for a browser string, fetched after the bot tests, shows that has not happened.
  • Check your logs for the status the real bot gets. The curl test sends the string from your own IP, so it proves the rule logic, not what a real GPTBot sees. See your page as an AI crawler explains that limit in more depth.

What can user agent matching not stop?

It stops honest bots, and only those. The header is whatever the client chooses to send, so a scraper can call itself Mozilla/5.0 and walk past every rule above. There is also a reverse problem: anything that sends GPTBot, including your own monitoring or a colleague's test, gets blocked.

If you need to treat real and fake crawlers differently, verify the source IP:

  • OpenAI publishes IP ranges as JSON, for example gptbot.json, and Anthropic publishes claude.com/crawling/bots.json. Anthropic says a request from an IP on its list comes from Anthropic.
  • A request that claims one of these bots from an IP outside the operator's list is not that bot. How to verify a real AI crawler shows the lookup and what to do with fakes.
  • Those lists change, so fetch them on a schedule instead of pasting ranges into a config.

Be careful about turning this into an IP-only block. Anthropic advises against blocking its crawlers by IP address, because it can keep them from reading robots.txt, so an IP block may not reliably guarantee an opt-out. The same logic applies to any rule that returns an error for /robots.txt. Under RFC 9309 a crawler that gets a 4xx for robots.txt may treat the file as unavailable and crawl without restrictions, so your opt-out is unreadable exactly when you most want it to be read.

If the problem is load, not permission, a 403 is the wrong tool. Rate limiting AI crawlers covers slowing a bot with 429 and per-bot budgets instead.

How do you keep search crawlers and Googlebot reachable?

Treat the block list as an allow-by-default system: only the tokens you name are refused. Then protect the pieces that break most often.

  1. Name full tokens, never fragments. GPTBot, not GPT. ClaudeBot, not Claude. No pattern containing just bot, crawler or ai.
  2. Keep /robots.txt outside every rule. Use the $is_robots_txt map in nginx and the second RewriteCond in Apache.
  3. Keep the search tokens out of the list. If you want to appear in ChatGPT search, OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing its published IP ranges, so do not match it. Perplexity says sites that use a WAF may need to explicitly allow its bots.
  4. Re-run the curl loop after every change, including after a plugin, a CMS update or a new hosting rule rewrites your .htaccess. A security plugin can also add its own bot rules.
  5. Look at real traffic after a day or two. Count the status codes for each token in your access log. Search bots should show mostly 200, and the blocked training bots should show 403.

At the time of writing these tokens match the operators' documentation. Operators add and rename agents, so recheck the lists before you copy a rule into a new site.

Frequently asked questions

Not if your patterns name only training tokens such as GPTBot and ClaudeBot. OpenAI says GPTBot and OAI-SearchBot are independent settings, and ChatGPT search eligibility comes from OAI-SearchBot. Anthropic separates ClaudeBot from Claude-SearchBot the same way. The risk is a pattern broad enough to catch a search bot.

Should I use robots.txt or a server rule for GPTBot?

Use robots.txt as the primary signal, because OpenAI and Anthropic document it as the way to opt out, and add the server rule if you want requests refused. They do different jobs, and nothing stops you from using both.

Can I block ChatGPT-User or Perplexity-User this way?

Yes, because the rule does not depend on the bot's cooperation. It also stops the person's request from reading your page, so ChatGPT or Perplexity cannot fetch it on their behalf. Decide whether that is what you want before adding the tokens.

Why does my rule block nothing?

Common causes: .htaccess is ignored because AllowOverride is None, RewriteEngine On is missing, the if sits in a location that does not serve the request, nginx was not reloaded, or a CDN answers from cache before the request reaches your server. Test against the origin directly first, then through the public hostname.

Check what your site returns to AI crawlers

After the rules are live, the free check reads one page the way AI crawlers do, with no account needed. It shows the robots.txt rule for each named AI crawler and whether the page responds normally, which helps confirm you did not block a search bot by accident. For a whole site, an audit covers up to 500 pages with evidence and prioritized fixes.

Filed under Access and robots.txt. Topics: ai-crawlers, robots-txt, bot-protection, nginx, apache