Skip to content

AI crawlers overloading your server: how to reduce the load

AI crawlers overloading your server? Measure which bots cost the most from logs, then cut load with caching, 429 and Retry-After, robots.txt scoping and blocks.

By Crawlito · 6 min read

In short

  • Measure before you block: count requests, bytes and server time per crawler token from your access logs, and verify that the heaviest bots are genuine.
  • Most crawler load lands on uncached, parameterized or search URLs, so fixing caching and crawl traps often removes more load than any bot rule.
  • A 429 or 503 with Retry-After slows a crawler down without refusing it; Google says to use these for hours or a day or two, not weeks.
  • robots.txt can keep training crawlers out entirely or keep any crawler away from expensive paths, but it does nothing against bots that ignore it.
  • Block unverified traffic that claims to be an AI crawler first, because fake crawlers ignore robots.txt and are the easiest load to remove.
On this page

If AI crawlers are overloading your server, start by measuring which bots cost the most and which URLs they hit, from your access logs. Then reduce the load in order of least collateral damage: cache what they fetch, close crawl traps, keep crawlers off expensive paths with robots.txt, slow the remaining heavy hitters with 429 and Retry-After, and block traffic that only pretends to be a known crawler. Blocking everything is the last step, not the first, because it can also remove you from AI search.

This guide covers the diagnosis and the order of fixes. For per-bot rate limit configuration, see rate limit AI crawlers without locking out AI search.

Step 1: confirm that crawlers are the cause

High load has many causes: a slow database query, a traffic spike, a cache that stopped working after a deploy. Before blaming bots, check whether the load lines up with crawler traffic.

Add request time to your access log so you can measure cost, not just counts. In nginx:

nginx
log_format timed '$remote_addr [$time_local] "$request" $status '
                 '$body_bytes_sent $request_time "$http_user_agent"';
access_log /var/log/nginx/access.log timed;

$request_time is the time nginx spent on the request, in seconds. On Apache, %D in LogFormat logs the time taken in microseconds.

Then look at requests per minute during the slow period:

bash
awk '{print substr($2, 2, 17)}' access.log | sort | uniq -c | sort -rn | head

If the busiest minutes are also the minutes when one user agent dominates, you have a crawler problem. If not, look elsewhere.

Step 2: measure cost per crawler, not just hits

A crawler that fetches ten thousand cached HTML pages costs less than one that fetches five hundred uncached search result pages. Sum request time and bytes by crawler token:

bash
awk -F'"' '{
  ua=$4; split($3, a, " "); bytes=a[2]; t=a[3];
  if (match(ua, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Googlebot|bingbot|meta-externalagent/)) {
    bot=substr(ua, RSTART, RLENGTH); n[bot]++; s[bot]+=t; b[bot]+=bytes
  }
} END { for (k in n) printf "%-20s %8d req %10.1f s %12d bytes\n", k, n[k], s[k], b[k] }' access.log | sort -k4 -rn

The output ranks crawlers by total server time. That is the number that matters for load. Our guide to AI crawlers in server logs lists the tokens to search for and how to count by day.

Next, find which URLs the expensive crawler spends its time on:

bash
grep "ClaudeBot" access.log | awk -F'"' '{split($2, r, " "); print r[2]}' \
  | sed 's/?.*/?.../' | sort | uniq -c | sort -rn | head -20

Collapsing query strings shows whether the load comes from a few templates with endless parameter combinations, which is the most common pattern.

Check the heavy hitters are real

A user agent string is not proof of identity. Before you change any rule, verify the top crawlers against the operator's published IP lists or reverse DNS, as described in how to verify real AI crawlers. If much of the "GPTBot" traffic comes from IPs outside OpenAI's published ranges, your problem is impostors, and robots.txt will not help.

Step 3: cache what crawlers fetch

Crawlers mostly request the same public pages that anonymous visitors see. If those pages are served from a cache, crawler traffic costs very little.

  • Cache anonymous HTML at the CDN or reverse proxy. Many CMS setups bypass the cache for any request with a cookie or query string. Crawlers rarely send cookies, but they do follow links with parameters.
  • Normalize or ignore tracking parameters in the cache key. ?utm_source= variants of the same page should not each hit your origin.
  • Support conditional requests. If your server sends ETag or Last-Modified and answers If-None-Match or If-Modified-Since with 304 Not Modified, a crawler that revalidates gets a tiny response instead of the full page. See ETag and Last-Modified for how to test this.
  • Serve static assets from the CDN. Rendering crawlers fetch CSS, JavaScript and images as well as HTML.

Check the cache hit rate for crawler requests specifically. If your proxy logs $upstream_cache_status (nginx) or the CDN adds a cache status header, filter it by user agent. A low hit rate for bots while human traffic is mostly cached usually means bots hit URLs your cache rules exclude.

Step 4: close crawl traps

Crawl traps are URL spaces with no practical end: calendars that page forward forever, faceted filters that combine into millions of URLs, internal search pages, session IDs in URLs. Crawlers that follow links can spend most of their requests there, and those pages are usually the most expensive to render.

Fixes, from best to least effective:

  1. Stop generating the links. Filters that change the URL can often use a form or a single canonical parameter order instead.
  2. Return 404 for invalid combinations rather than an empty result page with 200.
  3. Keep crawlers out with robots.txt if the URLs must exist for people but have no value in search:
User-agent: *
Disallow: /search
Disallow: /*?filter=
Disallow: /calendar/

Remember that a named group, such as User-agent: GPTBot, does not inherit these * rules, so repeat them in every named group. Faceted navigation crawl traps goes deeper on filters.

Step 5: slow down, do not refuse, the crawlers you want

For crawlers you want to keep, such as Googlebot, Bingbot, OAI-SearchBot or PerplexityBot, slow them down rather than block them.

Google's guide to reducing Googlebot's crawl rate says to return 500, 503 or 429 instead of 200 for a short period, such as a couple of hours or one to two days, and that you can add a Retry-After header to a 503 or 429. It warns against doing this for longer than one to two days, because URLs that keep returning errors for several days may be dropped from the index. If serving errors is not practical, Google offers a special request form to report an unusually high crawl rate.

Other operators document less. Crawl-delay in robots.txt is not part of RFC 9309. Bingbot, ClaudeBot and CCBot document support for it, while Google ignores it and OpenAI and Perplexity do not mention it. Check each operator's crawler page before relying on it.

Never throttle /robots.txt itself. Google treats a 429 on robots.txt as a server error and pauses crawling of the site while it retries.

Step 6: decide what to block

Once caching, traps and throttling are in place, look at the remaining load and decide by crawler purpose:

Crawler typeExamplesIf you block it
Fake crawlers (fail verification)Anything claiming a known token from the wrong IPsNothing of value is lost
Training crawlersGPTBot, ClaudeBot, CCBotContent is not used for future model training; no documented effect on AI search
Search crawlersOAI-SearchBot, Claude-SearchBot, PerplexityBotPages can drop out of that assistant's search answers
User-triggered fetchersChatGPT-User, Claude-User, Perplexity-UserUsers who ask about your page get an error

Training crawlers are the usual candidates when load is the main concern, because blocking them does not affect search visibility according to the operators' documentation. OpenAI, for example, says GPTBot and OAI-SearchBot settings are independent. A robots.txt that does this:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

For bots that ignore robots.txt, or for unverified traffic using a known token, block at the server or CDN. Block AI crawlers in nginx, Apache and .htaccess has the syntax. Prefer a 403 or a plain connection drop for clear abuse, and a 429 for crawlers you only want to slow.

Step 7: watch the effect

After each change, rerun the cost-per-crawler report for a comparable period. You are looking for three things:

  • Total server time from crawlers is down.
  • The crawlers you want are still getting mostly 200 and 304 responses, not 403 or 429.
  • /robots.txt returns 200 to every crawler.

Changes to robots.txt are not instant. Google caches robots.txt for up to 24 hours in most cases, and OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update.

Frequently asked questions

Which AI crawler uses the most server resources?

It depends on the site and changes over time. Measure server time per user agent from your own logs rather than relying on published rankings, and verify the top entries are genuine before acting.

Blocking GPTBot stops OpenAI's training crawler. OpenAI documents that OAI-SearchBot, its search crawler, is controlled separately, so ChatGPT search access depends on the OAI-SearchBot rules.

Is returning 429 to Googlebot safe?

For a short period, yes. Google recommends 500, 503 or 429 to reduce crawling urgently, but warns that doing it for more than one to two days can lead to URLs being dropped from the index.

Does caching help if crawlers request pages nobody else visits?

Partly. Deep pages may not be in cache when a crawler asks for them, but conditional requests, cached assets and fixed crawl traps still reduce the cost of each request.

Get a second view of your crawler access

Load problems often hide access problems: a firewall rule added in a hurry can start blocking the search crawlers you need. To see how one of your pages responds to AI crawlers today, run the free check.

Filed under CDN and bot protection. Topics: server-logs, rate-limiting, caching, bot-protection, ai-crawlers