Skip to content

robots.txt file size limit: 500 KiB and what gets ignored

Google ignores robots.txt content past 500 KiB, and RFC 9309 sets the same minimum for crawlers. See what that drops, how to measure your file and how to shrink it.

By Crawlito · 8 min read

In short

  • Google enforces a robots.txt size limit of 500 KiB (512,000 bytes) and ignores content after it, so rules near the end of a very large file may never apply.
  • RFC 9309 says a crawler's parsing limit must be at least 500 kibibytes, so other crawlers may read more than Google does, but a crawler that follows the standard may not set its limit lower.
  • OpenAI's crawler page and Anthropic's help article on its crawlers do not state a robots.txt size limit at the time of writing, so keep the whole file under 500 KiB to be safe.
  • Google ignores invalid lines, including a Unicode byte order mark, and it ignores fields other than user-agent, allow, disallow and sitemap, such as crawl-delay.
  • The usual cause of a huge file is per-URL rules and the same block list copied into every named group; merged User-agent lines and path prefixes shrink it without changing what it blocks.
On this page

The robots.txt size limit that Google documents is 500 KiB, which is 512,000 bytes. Google ignores any content after that point. RFC 9309, the robots.txt standard, says a crawler's parsing limit must be at least 500 kibibytes, so Google's figure is the floor the standard sets, not a quirk of one search engine. If your file is under 500 KiB, size is not your problem. If it is over, the rules at the bottom may not apply at all.

Most files never get near this. The cases that do are generated ones: long AI bot block lists copied into several groups, a plugin that writes a line per URL, or a template that appends rules on every deploy. This post covers what the limit is, what happens past it, how to measure your file the way a crawler receives it, how to shrink it, and the format rules that silently drop rules even in small files.

What is the robots.txt file size limit?

Two primary sources give the same number.

  • Google: "Google enforces a robots.txt file size limit of 500 kibibytes (KiB)," and content after the maximum file size is ignored. This is in Google's robots.txt specification.
  • RFC 9309: section 2.5 says crawlers should impose a parsing limit to protect their systems, and that the limit must be at least 500 kibibytes. The text is in RFC 9309.

A kibibyte is 1,024 bytes, so 500 KiB is 512,000 bytes. A 500 KB file (500,000 bytes) is just under it; a file that a tool reports as 520 KB is over it.

The RFC sets a minimum, not a maximum. A crawler may read more than 500 KiB. What the standard does not allow is a limit below it. The practical reading is that the 500 KiB figure is the safe ceiling for any crawler that follows the standard.

Chrome's Lighthouse has an audit for this too. Its robots.txt is not valid page tells you to keep the file under 500 KiB and warns that search engines may stop processing it midway through if it is larger.

What about AI crawlers?

At the time of writing, OpenAI's crawler overview and Anthropic's help article on its crawlers do not state a robots.txt size limit. That does not mean there is none. It means you cannot look one up, so treat 500 KiB as the working limit for every crawler. If you want the file to behave the same way everywhere, keep it comfortably below that.

What happens to rules after the limit?

Google says content after the limit is ignored. For you, that means rules past byte 512,000 are not applied, as if they were not in the file.

That matters most for Disallow rules. If a block for a crawler sits near the end of an oversized file and is cut off, the crawler sees no rule that stops it, so the paths you meant to protect are open to it. The failure is easy to miss: the file looks right when you open it, because the cut happens on the crawler's side.

Two details are not documented, so do not assume them:

  • Google's page does not say whether the limit applies to the compressed or the decompressed file.
  • It does not say how a rule that straddles the cut is handled.

Because of that, measure the decompressed file and aim to be well under the limit, not just under it. Rules that matter most, such as the blocks for the crawlers you most want to control, are safer near the top of the file.

How to measure your file the way a crawler receives it

Measure what your server returns, not the file in your repository. A CDN, a plugin or a rewrite rule can serve something different from what you edited.

Count the bytes of the response body:

bash
curl -sL https://example.com/robots.txt | wc -c

Compare the result with 512,000. To see how much headroom you have:

bash
curl -sL https://example.com/robots.txt | wc -c | awk '{ printf "%d bytes, %.1f KiB, %.0f%% of 500 KiB\n", $1, $1/1024, $1/5120 }'

Then check the status code and the content type, because a 5xx on robots.txt is a bigger problem than size. Under RFC 9309 a crawler must assume complete disallow when the file is unreachable with a 5xx, and Google says it stops crawling for the first 12 hours while it keeps retrying. Our guide to testing robots.txt against each AI crawler shows the full routine.

bash
curl -sIL https://example.com/robots.txt

Lighthouse does not check that the file is at the root of the host, only that its contents are valid, so confirm the location yourself. Google's page says the file must sit in the top-level directory of a site.

Why robots.txt files grow

A file reaches hundreds of kilobytes only when something is generating lines. As an illustration, one User-agent line plus a Disallow: / line is roughly 30 bytes, so a hand-written list of every AI crawler you know about is a few kilobytes. These are the usual causes of the rest:

  • Per-URL rules. A tool or plugin that writes one Disallow line for every URL, tag page, parameter or retired product keeps growing as the site does.
  • The same list in every group. A crawler follows only the group that names it and ignores User-agent: *, as explained in robots.txt groups. So people copy the whole shared rule set into each named group, and the file grows with every crawler added.
  • Appended deploys. A build step that adds rules without replacing the old ones leaves duplicates.
  • Long comments. Comments are ignored by parsers but still count toward the size.

How to shrink a robots.txt file

Shrink it without changing what it blocks, then verify. Three changes do most of the work.

Merge groups that share rules

Several User-agent lines in a row share the rules below them, which is the standard way to give similar crawlers one policy. This block:

User-agent: GPTBot
Disallow: /private/

User-agent: ClaudeBot
Disallow: /private/

User-agent: CCBot
Disallow: /private/

becomes:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /private/

Also merge separate groups for the same crawler. RFC 9309 says the groups that match a crawler are combined, so a second User-agent: GPTBot block is redundant if it can live in the first. The robots.txt for AI crawlers guide has templates you can start from.

Use path prefixes and wildcards

A rule matches every URL that starts with its path, so one prefix replaces a long list. RFC 9309 defines * as zero or more of any character and $ as the end of the match pattern.

# Before: one line per file
Disallow: /downloads/report-2021.pdf
Disallow: /downloads/report-2022.pdf
Disallow: /downloads/report-2023.pdf

# After: one rule
Disallow: /downloads/

Lighthouse recommends the same approach and gives the example of one pattern for all PDFs instead of listing each one. Check that the prefix does not catch URLs you want crawled before you ship it.

Move generation out of robots.txt

If a plugin writes thousands of lines, fix the source. Block a whole section or URL pattern, or put the exclusion where it belongs, for example a noindex response header or a meta tag on pages that should stay out of search. robots.txt vs meta robots explains the difference. Google's own advice for an oversized file is to consolidate rules, for example by moving excluded material into a separate directory so one rule covers it.

After you deploy the smaller file, re-measure it, then re-run your rule checks for each crawler you care about. Crawlers cache robots.txt: RFC 9309 says a cached copy should not be used for more than 24 hours unless the file is unreachable, and Google says it generally caches for up to 24 hours. How long robots.txt changes take covers the timing.

Format rules that also drop rules

A file can be tiny and still lose rules. These come from Google's specification and RFC 9309.

ProblemWhat happens
Not UTF-8RFC 9309 says the file must be UTF-8. Google says it may ignore characters outside the UTF-8 range, which can make rules invalid.
Byte order mark at the startGoogle ignores it as an invalid line. RFC 9309 does not mention it, so other parsers are not covered by the standard.
Unsupported fieldsGoogle supports user-agent, allow, disallow and sitemap, and says other fields such as crawl-delay are not supported. See Crawl-delay for AI bots.
Rule with no pathGoogle ignores it.
allow or disallow before any user-agentRFC 9309 says crawlers should ignore rules that are not in any group.
Unknown or invalid lineGoogle ignores invalid lines and uses the valid ones. RFC 9309 says crawlers must try to parse each line and use the parseable rules.
HTML instead of rulesGoogle tries to extract rules from it and ignores the rest.

Line endings are not a problem. Google accepts CR, CR/LF and LF, and the RFC's grammar allows the same three.

Lighthouse reports several of these as errors: no user-agent specified, a pattern that does not start with / or *, an unknown directive, an invalid sitemap URL, and $ used anywhere except the end of a pattern.

To check the encoding and the first bytes of your own file:

bash
curl -sL https://example.com/robots.txt -o robots.txt
head -c 3 robots.txt | xxd
iconv -f UTF-8 -t UTF-8 robots.txt > /dev/null && echo "valid UTF-8"

If the first three bytes are efbbbf, the file starts with a byte order mark. Remove it by saving the file as UTF-8 without BOM. For more of the mistakes that quietly change what a file does, read robots.txt mistakes that block AI crawlers.

Frequently asked questions

What is the maximum size of a robots.txt file?

Google documents 500 KiB (512,000 bytes) and ignores content after it. RFC 9309 says a crawler's parsing limit must be at least 500 kibibytes, so that is the safe size to stay under for any crawler.

Does Google read the whole file if it is over 500 KiB?

No. Google says content after the maximum file size is ignored, so rules past that point do not apply. Its documentation does not say how it treats a rule that is cut in half at the limit.

Do OpenAI and Anthropic crawlers use the same limit?

Neither OpenAI's crawler overview nor Anthropic's help article states a robots.txt size limit at the time of writing. RFC 9309 sets a minimum of 500 kibibytes for crawlers that follow it, so staying under that is the safest course.

Does a byte order mark break robots.txt?

Google ignores it as an invalid line, so it does not break Google's parsing. RFC 9309 does not address byte order marks, so save the file as UTF-8 without one rather than relying on every crawler to skip it.

Check what crawlers actually receive

Measure the live file first, then confirm each AI crawler token still gets the rules you intended. The free check fetches one page and its robots.txt with no account, and an audit covers the whole site.

Filed under Access and robots.txt. Topics: robots-txt, google, ai-crawlers, troubleshooting