robots.txt returns 404, 403 or 500: what crawlers do next
What crawlers do when robots.txt returns 404, 403, 5xx or a redirect, per RFC 9309 and Google's docs, plus how to check the status your file really returns.
By Crawlito · 7 min read
In short
- Under RFC 9309 a 4xx response for robots.txt means no rules apply, so a 403 or 404 allows a crawler to fetch everything.
- A 5xx or network error means the crawler must assume complete disallow, so a broken robots.txt can stop crawling of the whole host.
- Google stops crawling for the first 12 hours of robots.txt server errors, then uses its last good copy for up to 30 days.
- Google treats 429 on robots.txt as a server error, not as a 4xx, and follows at least five redirect hops before treating the file as missing.
- The status code your server sends to curl from your laptop may differ from what a firewall or CDN sends to a crawler, so test with crawler user agents too.
On this page
- What the standard says for each robots.txt status
- robots.txt returns 404 or 410: everything is allowed
- robots.txt returns 403 or 401: your rules are gone
- robots.txt returns 500 or 503: crawling stops
- robots.txt redirects: usually fine, sometimes not
- How to check what your robots.txt really returns
- Frequently asked questions
- Check what crawlers receive
When robots.txt returns a 404 or another 4xx error, crawlers treat the file as missing and may crawl everything. When it returns a 5xx error or the connection fails, the standard tells them to assume everything is disallowed. That asymmetry is the whole story: a missing file opens your site, a broken one closes it.
Both outcomes are usually accidents. A firewall answers /robots.txt with a 403 and silently removes every rule you wrote, or a deploy breaks the route and returns 500 and crawling stops. This guide covers what RFC 9309 and Google's robots.txt specification say for each case, the real-world causes behind each status, and how to check yours.
What the standard says for each robots.txt status
RFC 9309, the Robots Exclusion Protocol, defines four outcomes when a crawler requests /robots.txt. Google's specification follows the same structure with more detail on timing.
| Response for /robots.txt | RFC 9309 | Google's documented handling |
|---|---|---|
| 2xx | Follow the parseable rules | Same; generally cached for up to 24 hours |
| 3xx | Follow at least five consecutive redirects; the file found applies to the original host | Follows at least five hops, then treats it as a 404 |
| 4xx | "Unavailable": the crawler may access any resource | All 4xx except 429 treated as if no robots.txt exists |
| 429 | Grouped with 4xx | Treated as a server error |
| 5xx | "Unreachable": assume complete disallow | Stops crawling for 12 hours, then uses the last good copy for up to 30 days |
| DNS failure, timeout, reset | Unreachable | Treated as a server error |
Our broader guide to status codes and redirects for AI crawlers has a short version of this table. Below is what each row means in practice.
robots.txt returns 404 or 410: everything is allowed
A missing robots.txt is normal and harmless if you have no rules to express. RFC 9309 says that when the file is unavailable, "the crawler MAY access any resources on the server." Google's Search Console robots.txt report describes the 404 case the same way: Google can still crawl all URLs when no file exists.
It becomes a problem when you think you have rules and you do not. Common causes:
- The file exists in your repository but the framework or host does not serve it from the root. Static site generators and Next.js apps often need it in a
public/folder. - You moved to a new domain or subdomain and only the old host serves the file. Each host and protocol needs its own robots.txt, as explained in robots.txt for subdomains, protocols and ports.
- A WordPress plugin or server rule removed the virtual file WordPress normally generates.
robots.txt returns 403 or 401: your rules are gone
This is the most counterintuitive case. People assume a 403 on robots.txt means "forbidden, so keep out". The standard says the opposite: a 403 is a 4xx, so the file is unavailable and the crawler may access anything. Google says it treats all 4xx errors except 429 as if a valid robots.txt did not exist, and adds that 401 and 403 should not be used to limit crawl rate.
So if your WAF or bot protection answers /robots.txt with a 403 for crawler user agents, every Disallow line you wrote stops applying to the crawlers that respect the file. Meanwhile the same protection probably blocks those crawlers from your pages too, so the net effect is confusing: rules disappear, pages are blocked, and nothing in your robots.txt explains what crawlers see.
Typical causes:
- A firewall rule that blocks a user-agent string everywhere, including
/robots.txt. - A bot challenge (JavaScript or CAPTCHA) applied to every path. On Cloudflare a challenge response carries a
cf-mitigated: challengeheader, documented in Cloudflare's challenge detection guide. - Basic authentication left on a production host after launch.
The fix is to exempt /robots.txt from blocking and challenges entirely. It is a public file by design, and every crawler needs to read it, including the ones you want to block.
robots.txt returns 500 or 503: crawling stops
RFC 9309 says that if the file is unreachable because of a server or network error, "the crawler MUST assume complete disallow." Google spells out the timing:
- First 12 hours: Google stops crawling the site but keeps trying to fetch robots.txt.
- Up to 30 days: Google uses the last good cached copy while retrying. If it has no cached copy, it assumes no crawl restrictions.
- After 30 days: if the site is otherwise available, Google behaves as if there is no robots.txt; if the site has wider availability problems, it stops crawling and checks robots.txt periodically.
Google also notes that 503 errors cause fairly frequent retries, and its guidance on pausing a site says plainly not to return 503 for robots.txt, because that "blocks all crawling". If you put a site into maintenance mode, keep robots.txt serving a 200. Our guide to 503 maintenance mode covers how.
Other operators do not publish this level of detail. For crawlers such as GPTBot, ClaudeBot or PerplexityBot, assume RFC 9309 behaviour: a 5xx means stay out.
Why 429 is different
A 429 Too Many Requests sits in the 4xx range, so a strict reading of RFC 9309 would treat it as "unavailable". Google does not: it treats 429 on robots.txt as a server error, which means it stops crawling rather than crawling freely. If you rate limit crawlers, exempt /robots.txt from the limit so one throttled request does not pause crawling of the whole host. See rate limiting AI crawlers for limits that leave it alone.
robots.txt redirects: usually fine, sometimes not
A redirect from http:// to https:// or from example.com to www.example.com is normal. RFC 9309 says crawlers should follow at least five consecutive redirects, even to a different host, and that the robots.txt found at the end applies to the host originally requested. Google follows at least five hops and then treats the file as a 404.
Redirects go wrong in two ways:
- Long chains or loops. Past five hops, the file counts as missing and every rule is dropped.
- Redirecting to an HTML page. Some platforms redirect unknown paths, including
/robots.txt, to the homepage. The crawler then receives HTML with a 200. It contains no valid robots.txt lines, so nothing is disallowed. The same happens when a single-page app answers every path, including/robots.txt, with its HTML shell.
Google also says it does not follow logical redirects for robots.txt, such as frames, JavaScript or meta refresh. Only HTTP redirects count.
How to check what your robots.txt really returns
Start with the status line and headers, not the body:
curl -sI https://www.example.com/robots.txtYou want HTTP/2 200 and content-type: text/plain. Then follow redirects and count the hops:
curl -sIL https://example.com/robots.txt | grep -iE "^HTTP|^location"Now repeat the request as crawlers. Firewalls often treat user agents differently, so a 200 for curl's default user agent proves little:
for ua in "GPTBot/1.1" "ClaudeBot/1.0" "PerplexityBot/1.0" \
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"; do
printf "%-20s " "${ua%%/*}"
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" -A "$ua" https://www.example.com/robots.txt
doneRead the results this way:
| What you see | What it means | Fix |
|---|---|---|
200, text/plain, your rules | Working as intended | Nothing |
200, text/html | Probably a homepage or app shell, so no rules apply | Serve a real file at the root |
| 403 or 401 for some user agents | Those crawlers see no rules | Exempt /robots.txt from WAF rules and challenges |
| 404 | No rules apply anywhere | Publish the file, if you meant to have rules |
| 429, 500, 502, 503 | Crawlers that follow the RFC stop crawling | Fix the server error or exempt the path from limits |
Requests from your own machine do not reproduce a crawler's IP reputation, so a WAF may still treat the real crawler differently. Two more sources close that gap:
- Your server or CDN logs. Filter for requests to
/robots.txtand group by user agent and status code. Our guide to AI crawlers in server logs has the commands. - Google Search Console's robots.txt report. It shows which robots.txt files Google found for the top 20 hosts of a Domain property, when each was last checked, and a fetch status such as "Fetched" or "Not Fetched - Not found (404)". After fixing an error you can request a recrawl from the file's menu. See Google's help page for details.
Frequently asked questions
Does a 403 on robots.txt block crawlers?
No. Under RFC 9309 and in Google's documented handling, a 403 is treated like a missing file, so no rules apply. If you want to block a crawler, serve robots.txt with a 200 and a Disallow rule for it, and use the firewall separately.
What happens if robots.txt is down for a few hours?
Crawlers that follow RFC 9309 assume complete disallow while the file returns 5xx. Google stops crawling for the first 12 hours, then falls back to its last good copy for up to 30 days.
Is it bad to have no robots.txt at all?
Not in itself. A 404 means crawlers may access everything, which is fine for a site with nothing to keep crawlers out of. It is only a problem if you expected rules to apply.
How quickly do crawlers notice a fixed robots.txt?
RFC 9309 says crawlers should not use a cached copy for more than 24 hours, and Google generally caches for up to 24 hours. Google may keep a copy longer when it cannot refresh it, and you can request a recrawl in Search Console.
Should robots.txt redirect from http to https?
Yes, that is normal. Crawlers follow at least five redirects, and the file at the end applies to the host originally requested. Keep the chain short and make sure it ends at a plain-text file, not an HTML page.
Check what crawlers receive
A robots.txt that returns the wrong status is easy to miss because the file looks correct in your editor. Run the free check on one of your pages to see whether crawlers can fetch it and your robots.txt, or book an audit to have the whole site checked with evidence for each finding.
Filed under Access and robots.txt. Topics: robots-txt, http-status, ai-crawlers, troubleshooting