Rate limit AI crawlers without locking out AI search

To rate limit AI crawlers without losing AI search visibility, slow them down instead of shutting them out. Return 429 Too Many Requests with a Retry-After header when a bot goes over its budget, give each class of crawler its own budget, and keep search and user-triggered bots on a generous one. Then check your logs to confirm that the bots that feed ChatGPT search, Perplexity and Google are mostly getting 200 responses.

The aim is narrow: protect your server and your bill, and keep the crawlers you want able to read your pages. A rate limit that is too tight, or that answers with the wrong status, quietly does the same damage as a block.

Why sites rate limit AI crawlers

Crawler traffic is not free. Every uncached page fetch costs CPU, database queries and bandwidth, and on serverless or metered hosting it costs money per request. AI crawlers add a lot of that traffic. In an analysis published in December 2024, Vercel and MERJ counted about 569 million GPTBot fetches and 370 million from Claude in one month across Vercel's network, against about 4.5 billion from Googlebot (Search and Gemini combined). The same study found that ChatGPT's crawler hit 404 pages on nextjs.org 34.82% of the time, so a large share of that load bought nothing for either side.

The usual triggers for a rate limit are:

• Load spikes. A crawler fetches hundreds of pages a minute and slows the site for real visitors.
• Cost. Bot traffic pushes a site over a bandwidth or function-invocation tier.
• Expensive URLs. Search results, filters and calendars generate endless URL variants that each hit the database.
• Abuse under a borrowed name. Scrapers send GPTBot or ClaudeBot in the user agent because some sites wave those names through.

None of these needs a block. They need a ceiling.

Throttling vs blocking: what each status code tells a crawler

The status code you return is the message. HTTP defines 429 and 503 as temporary conditions, and Google documents that its crawlers slow down and come back when they see them. A 403 is a refusal. A challenge page is either a refusal or, worse, gets read as the content of your page.

429 is defined in RFC 6585, which says the response may include a Retry-After header. RFC 9110 defines Retry-After as either a number of seconds or an HTTP date. Google's guide to reducing crawl rate says you can return 500, 503 or 429 to slow Googlebot, and that errors on a significant number of URLs reduce the crawl rate for the whole hostname. It also warns that this is for a short period, such as a couple of hours or one to two days, and that crawling picks up again on its own once the errors drop. If Googlebot sees these codes on the same URL for several days, that URL may be dropped from Google's index.

A challenge page is the most common mistake. Legitimate crawlers do not solve challenges or CAPTCHAs (Anthropic, for one, says it will not try to bypass CAPTCHAs), so a rate limit rule set to "challenge" turns into a block for every bot it catches. Why a WAF challenge page means AI crawlers read nothing covers how to spot one. For status codes beyond rate limits, see status codes, redirects and AI crawlers.

Never rate limit robots.txt

Exempt /robots.txt from every limit. Google's robots.txt specification treats every 4xx on robots.txt as "no restrictions" except 429, which its status code guide counts as a server error. When Google cannot fetch robots.txt because of a server error, it stops crawling the site for the first 12 hours while it retries, then falls back to the last good copy. Crawlers that follow RFC 9309 strictly go the other way, because the RFC files every 4xx under "unavailable", which allows access to everything. A throttled request for robots.txt can either pause your crawl or erase your rules.

Should you use Crawl-delay instead?

Only as a supplement. Crawl-delay is not part of RFC 9309, and support is split. The crawl-delay guide goes into detail; at the time of writing the documented positions are:

Because so many crawlers ignore it, Crawl-delay cannot protect a server on its own. Enforce limits at the server or edge, and use Crawl-delay as a polite hint for the crawlers that read it.

How to set per-bot budgets in nginx

nginx's limit_req module uses a leaky-bucket limit keyed on any variable. Two details make per-bot budgets possible. First, requests with an empty key are not counted, so a map that returns "" for ordinary visitors leaves them unlimited by that zone. Second, if the key is the bot's name rather than its IP, all of a crawler's IP addresses share one budget, which matches how crawlers spread requests across many addresses.

This config gives each training crawler a tight budget, gives each search crawler a generous one, leaves user-triggered fetchers such as ChatGPT-User alone, and answers over-budget requests with 429 and Retry-After:

map $http_user_agent $ai_training_bot {    # all of this goes in http {}
    default               "";
    ~*GPTBot              gptbot;
    ~*ClaudeBot           claudebot;
    ~*CCBot               ccbot;
    ~*Bytespider          bytespider;
    ~*meta-externalagent  meta-externalagent;
}

map $http_user_agent $ai_search_bot {
    default               "";
    ~*OAI-SearchBot       oai-searchbot;
    ~*Claude-SearchBot    claude-searchbot;
    ~*PerplexityBot       perplexitybot;
}

limit_req_zone $ai_training_bot zone=ai_training:1m rate=30r/m;
limit_req_zone $ai_search_bot   zone=ai_search:1m   rate=5r/s;
limit_req_status 429;

server {
    listen 80;
    server_name example.com;

    # No limit_req here, so robots.txt is never throttled
    location = /robots.txt {
        try_files $uri =404;   # or proxy_pass, if your app generates it
    }

    location / {
        limit_req zone=ai_training burst=10 nodelay;
        limit_req zone=ai_search   burst=20 nodelay;
        error_page 429 = @too_many;
        # ... your proxy_pass or try_files
    }

    location @too_many {
        add_header Retry-After 120 always;
        return 429 "Too many requests. Retry later.\n";
    }
}

Notes on the choices:

• The rates are examples. Set them from your own logs: look at each bot's normal requests per minute and allow headroom above that. The search zone should rarely trigger.
• limit_req inheritance does the robots.txt exemption. nginx inherits limit_req from an outer level only when the current level has none, so keeping the directives inside location / leaves location = /robots.txt unlimited.
• always matters. Without it, add_header skips error responses, and the Retry-After header never goes out.
• Try it in dry-run first. limit_req_dry_run on; (nginx 1.17.1 and later) counts excess requests without rejecting them, so you can see what a limit would catch before it bites.
• Know where the boundaries are. OAI-SearchBot and GPTBot are separate tokens, so OpenAI's search crawler never falls into the training zone. If you want to refuse training crawlers outright rather than slow them, use robots.txt instead; the three crawler types explains which tokens control what.

Stop fake bots from spending the budget

Keying on the user agent has one weakness: anyone can send OAI-SearchBot in a header. A scraper using that name would draw down the search budget, and the real crawler would start getting 429s. The fix is to share the bot budget only with requests from verified IP ranges, and to key everything else by client IP. Generate a geo include file from the operators' published lists (OpenAI's searchbot.json, Perplexity's perplexitybot.json, Anthropic's bots.json) on a schedule, because those files change:

geo $verified_ai_ip {
    default 0;
    include /etc/nginx/ai-crawler-ips.conf;   # lines like "203.0.113.0/24 1;"
}

map "$verified_ai_ip:$ai_search_bot" $ai_search_key {
    "~^[01]:$"          "";                    # not a search bot: not counted
    "~^1:(?<bot>.+)$"   $bot;                  # verified: shared bot budget
    default             $binary_remote_addr;   # claims the name, unverified: per-IP
}

limit_req_zone $ai_search_key zone=ai_search:10m rate=5r/s;   # replaces the earlier ai_search zone

geo checks $remote_addr. If nginx sits behind a CDN or load balancer, set up the realip module first, or every request will appear to come from the proxy.

How to verify a real AI crawler has a script for checking IPs against those JSON files.

CDN and WAF rate limiting in general terms

Most CDNs and WAFs offer rate limiting rules with the same parts: a match (path, user agent, verified-bot status), a counting characteristic (usually client IP), a period, a threshold and an action. The same principles apply whichever vendor you use:

• Pick a block action that returns 429, not a challenge. Cloudflare's rate limiting rules return 429 by default when the action is Block, and let you set a custom status between 400 and 499. Its Managed Challenge, Non-Interactive Challenge and Interactive Challenge actions are challenges, and crawlers will not pass them.
• Remember what you are counting. On Cloudflare's Free and Pro plans, rate limiting counts by IP only. A per-IP limit rarely catches a crawler that spreads requests across many addresses, and it can catch several unrelated users behind one NAT.
• Exclude robots.txt from every rule, for the reasons above.
• Scope rules to expensive paths such as /search or filtered listings, rather than the whole site.
• Check what runs before your rule. Cloudflare's Bot Fight Mode cannot be bypassed with WAF custom rules, so a careful rate limit downstream of it does not help a bot that Bot Fight Mode has already challenged. Check your bot settings as well as your rate limiting rules.

Cut origin load before you limit anything

Often the cheapest rate limit is one you never need. Ways to make each crawler request cost less:

1. Cache HTML at the edge for anonymous requests. A crawler fetch served from cache costs your origin nothing.
2. Support conditional requests. Google's crawler overview says its crawling infrastructure supports ETag with If-None-Match and Last-Modified with If-Modified-Since, and recommends ETag. A 304 Not Modified is far cheaper than a full render.
3. Fix broken URLs and redirect chains. Every 404 a crawler requests is load with no value for either side. A cleaner internal link graph means fewer wasted fetches.
4. Stop generating crawlable URL variants. Keep filter and sort parameters out of plain links, or disallow them in robots.txt.

How to check that search bots are not being throttled

A limit you never measure is a guess. Log the result of every limit_req decision alongside the user agent. The $limit_req_status variable (nginx 1.17.6 and later) reports PASSED, DELAYED or REJECTED:

log_format crawlers '$remote_addr [$time_local] "$request" $status '
                    '$limit_req_status "$http_user_agent"';
access_log /var/log/nginx/crawlers.log crawlers;

Then count rejections per bot:

for bot in OAI-SearchBot Claude-SearchBot PerplexityBot Googlebot bingbot GPTBot ClaudeBot; do
  total=$(grep -c "$bot" /var/log/nginx/crawlers.log)
  rejected=$(grep "$bot" /var/log/nginx/crawlers.log | grep -cE ' REJECTED(_DRY_RUN)? ')
  echo "$bot: $rejected of $total requests rejected"
done

In dry-run mode the variable reads REJECTED_DRY_RUN, which the pattern above also counts. What a healthy result looks like: search crawlers and Googlebot close to zero rejections, training crawlers rejected only during bursts. If a search bot shows a steady share of 429s, raise its budget. Google Search Console's Crawl Stats report also groups Google's crawl requests by response type, for root-level properties. For a fuller log workflow, see how to find AI crawler visits in your server logs.

Frequently asked questions

Does returning 429 hurt my Google rankings?

Google says returning it for a short period, a couple of hours or one to two days, slows crawling and that the crawl rate recovers on its own once the errors stop. It warns that doing this for longer may affect how the site appears in Search, and that URLs returning these codes for several days may be dropped from the index.

Do AI crawlers honor Retry-After?

At the time of writing, OpenAI, Anthropic and Perplexity do not document how their crawlers treat Retry-After. Google's crawl-rate guide says you can include it with a 429 or 503 to tell its crawlers when to retry. Send it anyway: it is standard HTTP and costs nothing.

Should I rate limit ChatGPT-User, Claude-User and Perplexity-User?

Be careful. These agents fetch a page because a person asked about it, usually one page at a time. A 429 most likely means the assistant answers without your page, so give them a generous limit or none.

Is 503 or 429 better for throttling crawlers?

Use 429 when a specific client is sending too many requests and 503 when the server itself is overloaded or down for maintenance. Google treats both as signals to slow down.

Check what crawlers get from your pages

Before you tune budgets, confirm your current setup is not already turning crawlers away. The free check tests a single page, with no account needed. To run deterministic checks across up to 500 pages, with evidence and prioritized fixes, request an audit.