Crawl-delay for AI bots: who honors it and what to use instead

Crawl-delay is a robots.txt line that asks a crawler to wait between requests, but it is not part of the robots.txt standard and support is patchy. Bingbot, ClaudeBot and CCBot document that they honor it. Googlebot, Applebot and Amazonbot say they do not, and OpenAI and Perplexity do not mention it at all. If AI crawlers are loading your server, the dependable tool is a server or CDN rate limit that returns 429 Too Many Requests with a Retry-After header, because it caps requests whatever a bot reads. Aim it at the bots that cause the load and never at robots.txt.

This post lists who honors Crawl-delay according to each operator's own documentation (read on 2026-11-08), shows a working nginx rate limit and a Cloudflare alternative, and explains how to slow crawlers down without dropping out of AI search.

Is Crawl-delay part of the robots.txt standard?

No. RFC 9309, the Robots Exclusion Protocol, defines user-agent, allow and disallow. It mentions other records such as Sitemap only as examples of lines crawlers may interpret, and it does not define Crawl-delay.

That has two practical effects:

1. Support is opt-in per operator. A crawler that ignores the line is not breaking the standard.
2. The number has no agreed meaning. It is commonly read as seconds to wait between requests, but no specification says so. Anthropic's and Bing's examples both use Crawl-delay: 1, and Common Crawl says a higher number slows CCBot down.

Google is explicit. Its robots.txt specification says it supports only the user-agent, allow, disallow and sitemap fields, and that other fields such as crawl-delay are not supported.

Which AI crawlers honor Crawl-delay?

This table only reflects what each operator publishes. "Not documented" means the official crawler page does not mention Crawl-delay, so you should assume it has no effect.

A few details matter when you act on this:

• Anthropic calls Crawl-delay a non-standard extension it supports, with the example User-agent: ClaudeBot and Crawl-delay: 1. The article does not say whether the same applies to Claude-SearchBot or Claude-User, so name them separately if you want to try.
• Bing says a robots.txt crawl-delay always takes precedence over the hourly crawl rate you set in Bing Webmaster Tools Crawl Control. If you use Crawl Control, a leftover Crawl-delay line silently overrides it.
• Apple says Applebot's crawl rate adjusts automatically when a site slows down or returns errors. That is the behavior a server-side limit relies on.
• Cloudflare lists "an AI Crawler that does not respect the crawl-delay directive in robots.txt" as an example of a breach that can remove a bot from its verified bots list. That is Cloudflare's policy, not a promise from any crawler operator.

How to write Crawl-delay so it actually applies

If you use Crawl-delay for the bots that honor it, put it in the group that bot will read. Under RFC 9309 a crawler obeys the group that names it and falls back to * only when no group matches. Bing states the same for Bingbot: once a Bingbot group exists, it ignores the * section. A Crawl-delay line in * therefore does nothing for a bot that has its own group.

User-agent: Bingbot
Crawl-delay: 1
Disallow: /cart/
Disallow: /account/

User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /cart/
Disallow: /account/

User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

The Disallow lines are repeated in each group on purpose. A named group replaces * for that bot, so leaving them out would open /cart/ and /account/ to Bingbot and ClaudeBot. How robots.txt groups work explains the rule in full.

Keep the value small. If a crawler reads Crawl-delay: 10 as ten seconds between requests, it can fetch at most 8,640 URLs a day. On a site with 50,000 URLs, a full pass would take almost six days, and updated pages wait their turn.

The alternative that works: 429 with Retry-After

A rate limit on your server or CDN does not depend on what a crawler chooses to read. When a bot exceeds the rate you set, the server answers 429 Too Many Requests instead of the page.

RFC 6585 defines 429 as "the user has sent too many requests in a given amount of time". The response may include a Retry-After header saying how long to wait, and it must not be stored by a cache, so a standards-compliant cache or CDN should not keep serving the error to other visitors.

HTTP/1.1 429 Too Many Requests
Retry-After: 60
Content-Type: text/plain

Rate limit exceeded. Retry after 60 seconds.

Google's guide to reducing Googlebot's crawl rate recommends exactly this: return 500, 503 or 429 instead of 200 for a short period, such as a couple of hours or one to two days, optionally with Retry-After on 503 and 429. It adds three caveats worth keeping in mind for any crawler:

• The slowdown applies to the whole hostname, including pages that still return content.
• Google does not recommend doing it for longer than one or two days.
• If Googlebot sees these codes on the same URL for multiple days, that URL may be dropped from Google's index.

Other AI operators do not document how they respond to 429. Apple's statement about errors is the closest. Treat 429 as a short-term brake and fix the underlying load, not as a permanent setting. Status codes, redirects and AI crawlers covers how each code is read.

Do not use 403 as a throttle. Google says 4xx codes other than 429 have no effect on crawl rate and tells site owners not to use 401 or 403 to limit it. A 403 signals that the page is unavailable rather than asking the crawler to slow down.

An nginx rate limit for AI crawlers

This config limits named AI crawlers to two requests a second each, keyed by crawler name so the limit holds even when a bot spreads requests across many IPs. Requests with an empty key, such as normal visitors, are not counted by limit_req.

The map and limit_req_zone lines go in the http block; the rest goes in your server block.

map $http_user_agent $ai_crawler {
    default                 "";
    ~*GPTBot                gptbot;
    ~*OAI-SearchBot         oai-searchbot;
    ~*ClaudeBot             claudebot;
    ~*Claude-SearchBot      claude-searchbot;
    ~*PerplexityBot         perplexitybot;
    ~*CCBot                 ccbot;
}

limit_req_zone $ai_crawler zone=ai_crawlers:10m rate=2r/s;

server {
    # Never rate-limit robots.txt.
    location = /robots.txt {
        try_files $uri =404;   # or your normal proxy_pass, with no limit_req
    }

    location / {
        limit_req zone=ai_crawlers burst=20 nodelay;
        limit_req_status 429;
        error_page 429 = @too_many;
        # ... your normal proxy_pass or try_files
    }

    location @too_many {
        add_header Retry-After 60 always;
        return 429;
    }
}

limit_req_status defaults to 503, so set it to 429 explicitly. The always flag is needed because nginx skips add_header on error responses otherwise. Start with a generous rate and burst, then tighten using your logs.

One weakness: user agents can be faked. A scraper pretending to be GPTBot would use up GPTBot's allowance and push the real crawler into 429s. If that happens, check the source IPs against the operator's published list, as described in how to verify a real AI crawler, and block the impostors separately.

Rate limiting at Cloudflare

Cloudflare's rate limiting rules are on every plan, with one rule on Free, two on Pro and five on Business at the time of writing. On Free, counting is by IP over 10 seconds, and the rule expression can use the path and the Verified Bot field. The Block action returns 429 by default, and you can set any code from 400 to 499.

Because Free and Pro count per IP only, a crawler that uses many addresses may never trip the rule. Business adds IP with NAT support, and Enterprise with app security can also count by headers, path, ASN and other characteristics. Rate limiting AI crawlers without locking out AI search walks through rule design, and Cloudflare's AI crawler settings covers the bot controls that often block crawlers before any rate limit runs.

Slowing crawlers without disappearing from AI search

The goal is less load, not less visibility. These rules keep the two apart:

1. Exempt robots.txt from every limit. Google treats a 429 on robots.txt as a server error, and on a 5xx or network error it stops crawling the site for the first 12 hours while it retries. RFC 9309 tells crawlers to assume a full disallow when robots.txt is unreachable because of a server error.
2. Throttle training crawlers first. GPTBot crawls for OpenAI model training, ClaudeBot collects content that may be used to train Anthropic's models, and CCBot builds Common Crawl's open dataset. Each sends its own user agent, so a limit keyed on GPTBot or ClaudeBot does not touch requests from OAI-SearchBot or Claude-SearchBot. OpenAI also says its GPTBot and OAI-SearchBot robots.txt settings are independent.
3. Go gently with search crawlers. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot feed the search results that AI answers draw on. A 429 that lasts for days can cost visibility: Google says a URL that keeps returning these codes for multiple days may be dropped from its index, and other operators do not document what they do.
4. Do not rely on robots.txt for user-triggered fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. They fetch pages when a person asks, so Crawl-delay is not a lever for them and a server-side limit is the only control that applies.
5. Fix the cause. A crawler hammering endless filter URLs, calendars or search result pages is usually better handled with a Disallow for those paths and clean canonicals than with a rate limit.

How to check what crawlers are doing

Start with your access logs. These commands assume the common combined log format, where field 4 is the timestamp and field 9 is the status code.

The first command lists the busiest minutes for AI crawlers; the second counts how many 429s each crawler received.

grep -E "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot|CCBot" access.log \
  | awk '{print substr($4, 2, 17)}' | sort | uniq -c | sort -rn | head

awk '$9 == 429' access.log \
  | grep -oE "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot|CCBot|bingbot|Googlebot" \
  | sort | uniq -c

Then test your limit and its header from outside. Replace the URL with your own:

UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
for i in $(seq 1 40); do
  curl -s -o /dev/null -w "%{http_code}\n" -A "$UA" https://www.example.com/
done | sort | uniq -c

curl -sI -A "$UA" https://www.example.com/robots.txt | head -1   # should be 200, never 429

If a CDN or bot filter sits in front of your server, it may block a request that claims to be GPTBot but comes from your own IP before your rate limit runs, so test against the origin or read the results with that in mind.

If Googlebot or Bingbot shows up in the 429 count, your limit is catching search crawlers too. How to find AI crawler visits in your server logs shows how to separate real crawlers from fakes before you change anything.

Frequently asked questions

Does Googlebot respect Crawl-delay?

No. Google's robots.txt specification says crawl-delay is not supported. To slow Googlebot, return 500, 503 or 429 for a short period, no longer than one or two days, or file a special request through the Googlebot report in Search Console, which Google says can take several days to evaluate.

Does ClaudeBot respect Crawl-delay?

Yes, according to Anthropic. Its support article says it supports the non-standard Crawl-delay extension and gives Crawl-delay: 1 under User-agent: ClaudeBot as the example.

Does GPTBot respect Crawl-delay?

OpenAI's crawler documentation does not mention Crawl-delay, so there is no documented support. Use a server or CDN rate limit if GPTBot or OAI-SearchBot is putting load on your site.

Should I use 429 or 503 to slow crawlers down?

Use 429 when the problem is one client's request rate and 503 when the whole server is overloaded. Google treats both as a signal to slow down, and both can carry a Retry-After header.

Will a high Crawl-delay hurt my visibility in AI search?

It can for the bots that honor it. A long delay limits how many pages a crawler fetches each day, so new and updated pages take longer to be crawled. Keep the value low and target the bots that cause the load.

Check your robots.txt and status codes

A Crawl-delay line in the wrong group, or a named group that drops your Disallow rules, is easy to miss by reading config. The free Crawlito check fetches one page and its robots.txt with no account, and reports the page's status code and the robots.txt rule that applies to each named AI crawler.