robots.txt looks correct but uses the wrong text encoding
Diagnose a robots.txt saved as UTF-16, inspect the downloaded bytes, convert the source safely, and verify the intended crawler rules after publication.
By Crawlito · 6 min read
In short
- robots.txt must be UTF-8 text; a readable editor preview does not establish the published encoding.
- Changing a charset header does not convert the bytes in an incorrectly saved file.
- After conversion, verify both allowed and disallowed example paths without changing the intended policy.
On this page
- Download the response that crawlers actually receive
- Compare the bytes with the protocol requirement
- Find the export step that changed the file
- Convert a copy and preserve the intended rules
- Publish the artifact and inspect its public bytes again
- Test the policy with representative paths
- Add a release check at the source of the defect
- Frequently asked questions
If robots.txt looks correct in an editor but a validator cannot read its rules, inspect the downloaded file's encoding. A UTF-16 export can contain familiar text separated by null bytes. Convert the source to UTF-8, publish the converted file, and test the intended rules again; changing user-agent groups will not repair an encoding problem.
This diagnosis concerns the representation of the file, not whether you should allow training or search crawlers. Preserve the site's existing policy while making it readable. A successful conversion should leave the meaning of every deliberate rule unchanged.
Download the response that crawlers actually receive
Save a fresh public GET response, including headers, before opening the file in an editor:
curl -sS -L -D robots-headers.txt -o robots-downloaded.txt \
'https://www.example.com/robots.txt'
file robots-downloaded.txt
xxd -l 64 robots-downloaded.txtThese commands inspect a local copy and do not change the website. Replace the example hostname with the exact public host you are investigating.
Check the final response status and body. An HTML challenge page, login screen, or hosting error requires a different repair, even if the URL ends in robots.txt. A file-type guess is useful evidence, but it should be considered alongside the bytes and decoded content.
Record the protocol and hostname as well. Fetching the apex domain does not test a separate blog subdomain. The robots.txt scope guide explains why those requests need their own observations.
Keep the unmodified download as the before sample. Some editors silently convert a file when saving, which can erase the evidence you were trying to understand.
Compare the bytes with the protocol requirement
RFC 9309 requires robots.txt to use UTF-8 and the text/plain media type. Google's robots.txt specification also documents UTF-8 processing.
A common UTF-16 little-endian export starts with a byte-order marker and shows zero bytes between ordinary Latin letters. That pattern is a diagnostic clue, not permission to delete every zero byte from arbitrary content.
A browser or editor may recognize the encoding and display readable text automatically. A crawler's robots parser is a different consumer. Do not use a screenshot of readable directives as proof that the underlying file meets the published contract.
Also distinguish a UTF-8 byte-order marker from UTF-16. They are different byte sequences and should not be described as the same defect. Check the actual encoding before choosing a conversion.
If the bytes already decode correctly as UTF-8, investigate syntax, grouping, response status, or stale delivery instead. Encoding is one testable hypothesis, not a universal explanation for a denied path.
Find the export step that changed the file
Trace the published artifact back through the release process. The source may be correct while a desktop editor, spreadsheet export, deployment script, or storage upload produces a different encoding.
Compare the repository file, build output, and public download. Identify the first boundary where their bytes differ. That comparison prevents a local repair from being overwritten by the next automated release.
For a generated file, inspect how the generator opens its output stream. Set the output encoding deliberately rather than relying on the machine's default. For a manually maintained file, save it as plain UTF-8 text and keep it in version control.
Avoid copying directives through a rich-text application. Formatting controls, smart punctuation, or an HTML wrapper can create additional problems unrelated to UTF-16.
Record who owns the publishing step. A one-time correction to an uploaded object may be useful for recovery, but the durable fix belongs in the process that recreates that object.
Convert a copy and preserve the intended rules
Once the input encoding is established, convert a copy. For a file confirmed to be UTF-16, a local conversion can look like this:
iconv -f UTF-16 -t UTF-8 robots-downloaded.txt > robots-utf8.txt
file robots-utf8.txtDo not run this command against a file merely suspected of using UTF-16. A mistaken input encoding can corrupt valid text. Keep the original and inspect the converted result before deployment.
Compare every user-agent group, path, comment, and sitemap reference. The converted file should express the same policy. If you also want to change the policy, review that as a separate edit with its own expected outcomes.
Do not replace the file with an empty allow-all version just to make the parser pass. That would combine a format repair with an access decision.
Remember that robots.txt is not access control for confidential material. Encoding changes should not be used as a substitute for the site's existing authentication and authorization controls.
Publish the artifact and inspect its public bytes again
Deploy through the normal publishing workflow, then repeat the original GET. Save the new response under a different local filename.
Check that the public body is the converted artifact rather than an old cache entry or another generated copy. Inspect cache headers and the configured delivery path if the response does not match the release.
A header such as text/plain with a UTF-8 charset label describes the intended representation; it does not transform UTF-16 bytes into UTF-8. Verify the body as well as the label.
If multiple services can serve robots.txt, identify which one answered the public request. A static file, application route, and edge rule can compete for the same path.
Repeat the test on each relevant hostname. A successful repair on www does not establish that a documentation or store subdomain publishes the same valid file.
Keep the after sample with the release reference so another person can reproduce the comparison without relying on an editor's display settings.
Test the policy with representative paths
Choose a crawler token and two concrete paths: one intended to be accessible and one intended to be disallowed. Evaluate the converted file using a parser that follows the applicable documented matching rules.
The robots.txt testing guide covers that policy check. The purpose here is to show that fixing encoding restored readable rules without accidentally changing their meaning.
Include a path containing a non-ASCII character if the file deliberately handles such paths. Do not invent an encoding transformation for the URL while repairing the text file; path matching has its own rules.
Keep expected and observed decisions in a short table. A generic “valid” result does not show that the important article path remains accessible.
A live file repair also does not prove every crawler has already refreshed its copy. Treat later crawl observations as a separate follow-up, and avoid promising an immediate indexing or citation change.
Add a release check at the source of the defect
Make the publishing process reject an unexpected encoding before deployment. A lightweight check should read the actual generated artifact, not only a template that the build later rewrites.
Have the check confirm that decoding succeeds and that the file still contains the intended groups. It should flag suspicious null bytes for investigation rather than silently deleting them.
Keep the test fixture small and readable. Include an ordinary directive, a comment, and a representative path so the check exercises the content the generator actually produces.
For a client handoff, report the original encoding, the source of the export error, the repaired artifact, and the policy examples that passed. This is stronger evidence than saying robots.txt was “optimized.”
If matching decisions still differ after conversion, review the robots.txt groups guide before changing access policy.
Frequently asked questions
Can I fix this by changing Content-Type alone?
No. Correct an inaccurate header where necessary, but an encoding label does not rewrite the response bytes.
Should I remove all byte-order markers automatically?
First identify the actual encoding and the parser's documented behavior. UTF-8 and UTF-16 markers are different, and indiscriminate byte deletion is not a safe conversion.
Does a readable robots.txt guarantee indexing?
No. It establishes a usable access-policy file. Discovery, page content, indexing decisions, and AI citations remain separate considerations.
Once the public file is readable, use the free Crawlito check on an important page to inspect its current access and page-level signals alongside the repaired rules.
Filed under Access and robots.txt. Topics: robots-txt, troubleshooting, checklist