robots.txt is not valid in Lighthouse: what each error means
Lighthouse says robots.txt is not valid? Each error message mapped to its cause, its fix, and whether crawlers are affected, including Content-Signal lines.
By Crawlito · 8 min read
In short
- Lighthouse's robots-txt audit fails on seven line-level messages, such as Unknown directive and No user-agent specified, and also on a 5xx response, which under RFC 9309 makes compliant crawlers assume the whole site is disallowed.
- RFC 9309 says crawlers must use the rules they can parse, and Google says it ignores invalid lines, so an Unknown directive failure usually means the audit failed, not that crawling was blocked.
- Lighthouse added content-signal to its directive safelist in version 13.0.2, released on 10 February 2026, so only older Lighthouse builds flag Cloudflare's Content-signal line.
- Rules placed before the first User-agent line are a real problem: Lighthouse reports No user-agent specified, and RFC 9309 says crawlers should ignore rules that are not in any group.
- A 4xx response or an empty robots.txt makes the audit not applicable rather than failed; from Lighthouse 13.6.0 a 429 response fails it like a 5xx.
On this page
"robots.txt is not valid" means Lighthouse found at least one line it could not validate, or could not fetch the file properly. Most of the line errors do not block crawlers, because RFC 9309 tells crawlers to use the rules they can parse and Google says it ignores invalid lines. Three cases do matter: a 5xx response, rules placed before any User-agent line, and a mistyped rule that some crawlers may skip. Expand the audit, read the Error column for each line, and use the table below to decide which kind you have.
The audit runs in Lighthouse's SEO category, in Chrome DevTools and in PageSpeed Insights. It checks the file for the whole host, not just the page you tested. This post follows the audit's own source code, robots-txt.js, because it lists more messages than the Chrome documentation page does. For mistakes that pass validation but still block AI crawlers, see robots.txt mistakes that block AI crawlers.
How the Lighthouse robots.txt audit decides
Lighthouse fetches /robots.txt and checks the response before it reads a single line:
| Response | Audit result | What Lighthouse shows |
|---|---|---|
| 5xx | Fail | Request for robots.txt returned HTTP status: 500 (or the code it got) |
| 429 (Lighthouse 13.6.0 and later) | Fail | The same message with 429 |
| 4xx other than 429 | Not applicable | Nothing to fix from Lighthouse's point of view |
| 200 with an empty body | Not applicable | Same |
| Fetch error | Fail | Fetch of robots.txt failed, followed by the error from Lighthouse 13.4.1 on |
| 200 with content | Validated line by line | 1 error found or N errors found, with a table of lines |
The 429 change arrived in the 13.6.0 release of 27 October 2026; earlier versions treat a 429 as not applicable. In the line-by-line check, Lighthouse strips anything after #, trims the line, skips empty lines, lowercases the field name and validates the value. The table it prints has three columns: Line #, Content and Error.
What each robots.txt error message means
These are the exact messages the audit code throws, at the time of writing:
| Error | Example line | Cause | Does it affect crawlers? |
|---|---|---|---|
Syntax not understood | Disallow /admin/ | No colon on the line | Depends on the crawler: Google's parser accepts a missing colon, others may drop the rule |
Unknown directive | Content-Usage: train-ai=n | Field name not on Lighthouse's safelist | Usually no: crawlers ignore fields they do not support |
No user-agent specified | Disallow: /tmp/ above the first User-agent | A rule outside any group, or an empty User-agent: value | Yes: RFC 9309 says crawlers should ignore rules not in a group |
Pattern should either be empty, start with "/" or "*" | Disallow: private/ | Path without a leading slash | Possibly: Google says the path must start with / |
"$" should only be used at the end of the pattern | Disallow: /*.php$?id= | $ in the middle of a path | Possibly: the meaning of a mid-pattern $ is not documented |
Invalid sitemap URL | Sitemap: /sitemap.xml | Relative or malformed URL | Yes, for discovery: Google requires a fully qualified sitemap URL |
Invalid sitemap URL protocol | Sitemap: sftp://example.com/s.xml | Scheme other than http, https or ftp | The sitemap is unlikely to be fetched |
The Chrome documentation names five of these. Syntax not understood and Invalid sitemap URL protocol appear only in the code.
Syntax not understood
The line has no colon, so Lighthouse cannot split it into a field and a value. A single one usually means a typo such as Disallow /admin/. Google's open-source parser has a special case for this: when the colon is missing and the line has exactly two words, it treats the space as the separator. RFC 9309's syntax requires the colon, so other crawlers may drop the rule. Fix it by adding the colon.
Dozens of these errors at once usually mean the server returned an HTML page instead of a text file. HTML lines such as <!DOCTYPE html> have no colon, and lines that do contain one, such as a link, show up as Unknown directive instead. Check what the server actually sends:
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/robots.txt
curl -s https://example.com/robots.txt | head -5You want 200, a text/plain content type and rules in the first lines.
No user-agent specified
This message has two triggers in the code. The first is a User-agent: line with no value. The second is an Allow or Disallow rule that appears before the first User-agent line:
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xml
User-agent: *
Disallow: /admin/Here /checkout/ belongs to no group. RFC 9309 says crawlers SHOULD ignore allow and disallow rules that are not in any group, so that block may never apply. This is one of the errors with a real crawling effect. Move the rule under the group it was meant for. A Sitemap line above the first group is fine: Lighthouse only checks group placement for Allow and Disallow, and the RFC says a sitemap record does not end a group. The robots.txt groups guide explains which group each crawler reads.
Pattern should start with "/" or "*", and the "$" error
Both errors are about Allow and Disallow values. Google's robots.txt documentation says the path value must start with / to designate the root, so Disallow: private/ may not match what you intended. Write Disallow: /private/, or Disallow: /*private/ if you meant any path that contains private/.
RFC 9309 defines $ as the end of the match pattern and * as zero or more characters. A $ anywhere else is not given a meaning by the RFC or by Google, so do not rely on it. robots.txt wildcards and pattern matching shows how to write these patterns.
Invalid sitemap URL
Lighthouse parses the value as an absolute URL. Sitemap: /sitemap.xml fails, and so does https//example.com/sitemap.xml with the colon missing. A mistyped scheme such as htps:// still parses as a URL, so it gets the Invalid sitemap URL protocol error instead. Google's documentation says the sitemap value must be a fully qualified URL, including the protocol and host, though it may be on a different host from the robots.txt file. Use the full URL:
Sitemap: https://example.com/sitemap.xmlUnknown directive: an audit failure, not a crawl block
This is the message people worry about most, and it usually matters least. Lighthouse checks every field name against a safelist in its source. At the time of writing, in the current version, it contains:
| Field | Note in the Lighthouse source |
|---|---|
user-agent, disallow | Standard |
allow, sitemap | Universally supported |
crawl-delay | Yahoo, Bing, Yandex |
clean-param, host | Yandex |
request-rate, visit-time, noindex, content-signal, license | Not officially supported, but used in the wild |
Anything else fails the audit. That is stricter than crawlers are. RFC 9309 says crawlers may interpret other records, that parsing them must not interfere with the defined records, and that crawlers must use the rules they can parse. Google's robots.txt specification says Google ignores invalid lines and uses only the valid ones. So a line such as Content-Usage: train-ai=n, from an IETF draft, fails Lighthouse without changing how Googlebot reads your Allow and Disallow rules.
The exception is a typo in a field you meant as a rule. Useragent: * and Dissallow: /tmp/ both fail as Unknown directive. Google's open-source robots.txt parser accepts a few common typos, including useragent, user agent, dissallow and site-map. The RFC only says crawlers may be lenient about misspellings, so other crawlers may skip those lines entirely. Spell the field names correctly so every crawler reads the rule.
Why Content-Signal lines fail in some Lighthouse versions
Cloudflare's managed robots.txt adds a line like this to the wildcard group, according to its managed robots.txt documentation:
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /Lighthouse added content-signal to its safelist in pull request #16767, which shipped in Lighthouse 13.0.2 on 10 February 2026. The release notes said it was expected in the DevTools of Chrome 146 and in PageSpeed Insights within two weeks. So:
| Lighthouse version | Content-signal line | License line |
|---|---|---|
| 13.0.1 and earlier | Unknown directive | Unknown directive |
| 13.0.2 to 13.5.0 | Passes | Unknown directive |
| 13.6.0 and later | Passes | Passes |
Lighthouse checks only the field name, so the use=reference value does not cause an error. Field names are lowercased first, so Content-Signal and Content-signal are treated the same. An agentmap field was added to the safelist on the main branch on 3 November 2026, and at the time of writing it was not yet in a published release. So in 13.6.0 an Agentmap: line, which Lighthouse's own Agentic Resource Discovery gatherer reads from robots.txt, still fails this audit as Unknown directive.
If you still see this failure, the tool running the audit is older than 13.0.2. That is common with a pinned Lighthouse CI version, an old lighthouse npm package, or an older Chrome. Check the version before you edit the file: the Lighthouse HTML report states it at the foot of the report, and the CLI prints it with lighthouse --version. Cloudflare itself says Google Search Console may report "Syntax not understood" for the line and that it has observed no impact on crawling or SEO from those reports.
Removing the line to satisfy an old audit also removes the preference it states, and on Cloudflare it means changing your AI crawler policies. Content signals in robots.txt explains what the line says and how to change it.
How to rerun the audit and confirm the fix
- Note the current Lighthouse version, so you know whether a Content-signal failure is a version issue.
- Fix the lines the table points to, then fetch the live file with
curl, not the copy in your repository. A CDN can add content at the edge. - Run only this audit from the command line:
npx lighthouse https://example.com/ --only-audits=robots-txt \
--output=json --output-path=./robots.json --chrome-flags="--headless"- Read the result:
jq '.audits["robots-txt"] | {score, scoreDisplayMode, displayValue, items: .details.items}' robots.jsonA score of 1 is a pass. null with scoreDisplayMode of notApplicable means a 4xx or empty file. Each item gives the line number, the content and the error.
Lighthouse validates syntax; it does not tell you whether a crawler may fetch a given URL. Passing the audit and blocking OAI-SearchBot by accident are fully compatible. Test real URLs against each crawler's group as shown in how to test your robots.txt against each AI crawler.
Frequently asked questions
Does "robots.txt is not valid" hurt my SEO?
The audit itself is a diagnostic, not a ranking signal. What matters is the cause: a 5xx response or rules outside any group can change crawling, while an unknown field is ignored by crawlers that follow RFC 9309.
Why does Lighthouse flag Content-Signal when PageSpeed Insights does not?
They may be running different Lighthouse versions. Content-signal has been on the safelist since Lighthouse 13.0.2, so a failure on that line points to an older build.
Does Lighthouse flag Crawl-delay?
No. crawl-delay is on the safelist in the source code. Google does not support it, but Lighthouse does not count it as an error.
Is a missing robots.txt a Lighthouse failure?
No. A 404 or other 4xx response makes the audit not applicable. Only a 5xx, and from version 13.6.0 a 429, fails it on status alone.
Check what crawlers actually get
A clean Lighthouse result means the syntax is valid, not that AI crawlers can reach your pages. Run the free check on an important page to see whether robots.txt and the server let the main AI crawlers fetch it, with no account needed.
Filed under Access and robots.txt. Topics: robots-txt, content-signals, cloudflare, troubleshooting