Status codes and redirects: what AI crawlers do with them

Crawlers decide what to do with a URL from its HTTP status code before they look at a single word of content. In Google's documented handling, a 200 gets read, a 301 or 308 gets followed and the target treated as the real address, a 4xx gets dropped, and a 5xx or 429 makes the crawler slow down. Redirect chains, soft 404s and error codes on robots.txt are the common ways a site that works fine in a browser still hands crawlers nothing.

Google documents its handling in detail. The AI operators (OpenAI, Anthropic, Perplexity) document which agents they run and how they treat robots.txt, but at the time of writing none of them publishes how its crawlers treat redirects or individual status codes. So the practical rule is to make every URL you care about answer cleanly: one redirect at most, then a 200 with the real content.

What each status code tells a crawler

Google's HTTP status code documentation is the most complete public description of how a large crawler reacts. Treat it as the reference behaviour: the AI operators publish nothing that suggests their crawlers are more forgiving.

A few details matter more than they look:

• 404 and 410 are handled the same way by Google. All 4xx codes except 429 lead to the same result. RFC 9110 says a 410 means the page is gone and the condition is likely to be permanent, which is useful to other clients, but do not expect it to work faster in Google.
• 401 and 403 do not slow crawlers down. Google says plainly not to use them to limit crawl rate. They just mark the URL as unusable.
• 304 is a reply to a question. A server should only send it when the request carried If-Modified-Since or If-None-Match. A server that returns 304 to plain requests is broken, and a crawler with no cached copy gets an empty response.
• Content on an error page is ignored. A helpful 404 template full of links still gets discarded by Google; the links help visitors, not indexing.

How to see every hop and status code with curl

curl prints the status line and headers for each hop, which is what a crawler sees at the HTTP level.

curl -sIL http://example.com/old-page

-I asks for headers only, -L follows redirects, and -s hides the progress bar. On a badly configured site the output looks like this:

HTTP/1.1 301 Moved Permanently
Location: https://example.com/old-page

HTTP/2 301
location: https://www.example.com/old-page

HTTP/2 301
location: https://www.example.com/new-page

HTTP/2 301
location: https://www.example.com/new-page/

HTTP/2 200
content-type: text/html; charset=utf-8

That is four hops to one page: http to https, bare domain to www, old path to new path, then a trailing slash. Each is a separate request.

For a one-line summary per URL, which is handy in a loop over a list of URLs:

curl -s -o /dev/null -L -w '%{http_code} %{num_redirects} %{url_effective}\n' \
  http://example.com/old-page

200 4 https://www.example.com/new-page/

Two checks catch the problems a plain curl misses.

1. Use GET, not only HEAD. Some servers and CDNs answer HEAD differently from GET. If the results look odd, rerun with curl -sL -D - -o /dev/null <url>, which makes a real GET and prints only the headers.
2. Send a crawler user agent. Firewalls and bot rules often treat browsers and bots differently. Repeat the request as a crawler would identify itself:

curl -sIL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://www.example.com/new-page/

If the browser request gets a 200 and the GPTBot request gets a 403, or a 200 that turns out to be a challenge page, the block is in your CDN or firewall, not in robots.txt. A request from your own machine does not come from the operator's published IP ranges, so an IP-based rule can still behave differently for the real crawler. Seeing your page as an AI crawler with curl covers checking the content as well as the headers, and why a WAF challenge page means crawlers read nothing covers the firewall side.

Why long redirect chains and loops cost you

Google's crawlers follow up to 10 redirect hops by default, and Googlebot generally follows 10 for web content, according to the same status code page; specific Google products may have different limits. OpenAI, Anthropic and Perplexity do not document a hop limit for their crawlers. You cannot know how patient each one is, so design for the least patient: one hop, or none.

Long chains cause three kinds of trouble:

• Wasted requests. Each hop is a full round trip. A crawler with a fixed budget for your site spends it on redirects instead of pages.
• Mixed signals. A chain that mixes 301 and 302 hops tells a search engine "permanent" and "temporary" at once. Google treats 302s as a weak signal and may keep the old URL.
• Dead ends. A chain that ends in a 404, or loops back on itself, gives the crawler nothing. curl reports a loop as Maximum (50) redirects followed; browsers show "too many redirects".

Loops usually come from two systems disagreeing. A typical case: the CDN forces https, the origin thinks the request arrived over http (because the CDN talks to it over http) and redirects back to https, and the two bounce forever. Another is a CMS that adds a trailing slash while a server rule strips it.

How to fix a redirect chain

Point every old variant straight at the final URL. For the common http, https, www and bare-domain mix, one nginx server block for the two http hosts and one for the bare https host does it:

server {
    listen 80;
    server_name example.com www.example.com;
    return 301 https://www.example.com$request_uri;
}

server {
    listen 443 ssl;
    server_name example.com;
    ssl_certificate     /etc/ssl/example.com.pem;
    ssl_certificate_key /etc/ssl/example.com.key;
    return 301 https://www.example.com$request_uri;
}

Now http://example.com/page reaches https://www.example.com/page in one hop instead of two. Then do the same for path changes: when a page moves twice, update the first redirect to point at the newest URL rather than stacking a second redirect on top. Finally, update internal links, canonical tags and sitemap entries to the final URL so crawlers stop discovering the old one. Canonical tags and duplicate URLs and internal links and broken links go into both.

Prefer server-side redirects to anything in the page. Google's redirects guide says it can only see a JavaScript redirect after rendering, which may fail, so it should be a last resort. Crawlers that do not run JavaScript will not follow it at all. A meta refresh is a fallback for when you cannot change the server; Google treats an instant one as permanent and a delayed one as temporary.

Soft 404s and redirects to login pages

A soft 404 is a page that returns 200 but whose content is an error message or almost empty. Google's status code documentation says Search Console reports a soft 404 when the content of a 200 page suggests an error. A crawler that does not compare content against an error template simply reads your "Sorry, this product is no longer available" message as the page. Common causes:

• Single-page apps that return 200 for every route and draw the 404 message in JavaScript.
• Out-of-stock or deleted items that keep their URL with no content.
• Search and filter pages with zero results.

The fix is to return a real 404 or 410 when a page is gone, or a 301 to a genuinely equivalent page when one exists. Sending every dead URL to the homepage gives the crawler a page that has nothing to do with what it asked for.

A redirect to a login page is the other trap. The member page answers 302 to /login, the login page answers 200, and the crawler has read your login form. To the crawler that is a successful fetch of a page with no content worth using, and nothing in the status codes says otherwise.

curl -s -o /dev/null -L -w '%{http_code} %{url_effective}\n' https://www.example.com/guides/setup

200 https://www.example.com/login?next=/guides/setup

If a page is meant to be public, that redirect is a bug, often a session check applied to the wrong route. If it is meant to be private, accept that crawlers cannot read it. Paywalls and login walls covers the middle ground of previews and partial content.

What robots.txt status codes mean to crawlers

The status code of /robots.txt itself changes how a crawler treats your whole host, and it is easy to get backwards. RFC 9309, the robots.txt standard, sets the rules:

Two consequences surprise people:

• A 403 or 404 on robots.txt allows everything. If a firewall returns 403 on /robots.txt to bots, you have not blocked them; under the RFC you have removed every rule you wrote.
• A 500 on robots.txt blocks everything. Under the RFC the crawler must assume complete disallow, and Google stops crawling for the first 12 hours. A broken robots.txt route, a plugin that crashes or an origin timeout behind a CDN can take a site out of crawling while every page still works in a browser. Google's robots.txt specification gives the full timeline for its crawlers.

Redirects on robots.txt need care too. RFC 9309 says a robots.txt reached through redirects is parsed and its rules applied to the host the crawler started on, so http://example.com/robots.txt redirecting to https://www.example.com/robots.txt is fine. A redirect to an HTML page or a login screen leaves the crawler with nothing it can parse. Google does not follow JavaScript or meta refresh redirects for robots.txt. Check the file the same way as a page:

curl -sIL -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://www.example.com/robots.txt

You want a short chain ending in 200 with content-type: text/plain. For what to put inside the file, see robots.txt for AI crawlers.

How 429 and 503 fit with AI crawler traffic

AI crawlers can send a lot of requests, and the tempting response is a hard block. Status codes give you a softer tool. Google treats 429 and 5xx responses as a signal to slow down, and gradually raises its crawl rate again once the server returns 2xx. For planned downtime, a 503 with a Retry-After header (defined in RFC 9110) is the standard way to say "come back later".

HTTP/1.1 503 Service Unavailable
Retry-After: 3600
Content-Type: text/html; charset=utf-8

Keep it temporary. Google drops URLs that keep returning server errors, and the AI operators do not document how long their crawlers retry, so a 429 or 503 that lasts for weeks behaves like a block. Do not use 403 for this: it never asks a crawler to slow down, it only marks the page unusable.

Frequently asked questions

How many redirects will AI crawlers follow?

OpenAI, Anthropic and Perplexity do not publish a limit at the time of writing. Googlebot generally follows 10 hops for web content, and RFC 9309 asks crawlers to follow at least five for robots.txt. Aim for a single hop from any old URL to its final address.

Is a 302 redirect bad for AI search?

Not if the move really is temporary. For a permanent move, use 301 or 308: Google treats them as a strong signal to use the new URL, while a 302 is a weak signal and the old URL may be kept.

Should I return 404 or 410 for deleted pages?

Either works. Google handles them the same way, removing the URL from its index and crawling it less often. 410 says more precisely that the page is gone for good, which some other clients may use.

Does a 404 on robots.txt block crawlers?

No. Under RFC 9309 a 4xx response on robots.txt means the file is unavailable and the crawler may fetch anything. A 5xx response is the one that makes crawlers assume the whole site is disallowed.

Why does my page work in a browser but return 403 to a crawler?

Usually a CDN or firewall rule matches the crawler's user agent or IP range. Repeat the request with curl and the crawler's user agent to confirm it, then adjust the bot rule rather than robots.txt.

Check your own URLs

Start with the URLs that matter most: the homepage, your top landing pages and /robots.txt. Run curl -sIL on each, once as a browser and once with a crawler user agent, and look for more than one hop, any 302 on a permanent move, and any 4xx or 5xx. If you would rather have it done for you, the free check reports the status code, redirects and robots.txt rules for one page, and a full audit runs the same checks across up to 500 pages with the evidence for each fix.