Skip to content

Page indexed without content: why Google saw an empty page

Page indexed without content means Google indexed the URL but could not read it. Reproduce what Googlebot received, find the layer that emptied it and fix it.

By Crawlito · 9 min read

In short

  • Google's Page indexing report says a page indexed without content is in the index, but Google could not read its content, and that this is not a robots.txt block.
  • Google names two likely causes: the page may be cloaked to Google, or it may be in a format Google cannot index.
  • URL Inspection's crawled page shows the HTML and HTTP headers Google stored at the last crawl, so compare it with a live test and with a curl request before changing anything.
  • Google decides a file's type from the Content-Type header, reads only the first 2 MB of an HTML file uncompressed, and processes nothing from a 204 No Content response.
  • An origin under load should answer 503 or 429, not an empty 200; Google ignores the body of 5xx responses but passes whatever a 200 contains on to indexing.
On this page

"Page indexed without content" means Google has the URL in its index but could not read anything on it. Google's Page indexing report help gives two likely reasons: the page may be cloaked to Google, or it may be in a format Google cannot index. It also says plainly that this is not a robots.txt block. Google does not list causes beyond those two, but both come down to the same question: what response did Googlebot get that your browser does not, such as an empty body, a challenge page, a wrong Content-Type or a truncated file?

The fix is to find out what Googlebot actually got and which layer produced it. This post shows how to reproduce the response, the layers worth checking and how to confirm the fix. If your problem is a JavaScript app that renders nothing without a browser, start with the empty HTML problem instead.

What Google means by this status

The status is a warning, not an error. At the time of writing, Google lists it in the Improve page experience table of the Page indexing report, which holds issues that did not stop a page from being indexed. Three things in Google's wording narrow the search:

  • The page is indexed. Google crawled it and kept the URL, so a noindex rule did not keep it out of the index.
  • The content could not be read. Whatever Googlebot received had no usable text, or Google could not parse it.
  • It is not robots.txt. That case has its own status, indexed though blocked by robots.txt.

Google's suggested next step is to inspect the URL and read the Coverage section. The useful work is in what you do with that inspection. The table below is a checklist of responses that fit Google's description, not a list Google publishes.

What Googlebot may have receivedWhere it can come fromFirst thing to check
A 200 with an empty or near-empty bodyOrigin under load, a broken template, an app that failed mid-requestCrawled page HTML in URL Inspection
A challenge or block page with a 200 statusWAF, bot protection or CDN ruleResponse headers and body in a curl test with a Googlebot user agent
A file Google treats as the wrong typeWrong or missing Content-Type headerContent-Type in the crawled page's HTTP response
A body with no text Google could keepHTML whose first 2 MB holds no main text, or text that only loads through blocked resourcesRaw response size and the page resources list

How to reproduce what Googlebot saw

URL Inspection gives you two different views, and comparing them is the quickest way to tell a past problem from a current one. Google's URL Inspection help describes both:

  1. Inspect the URL and open View crawled page. This comes from the last crawled version of the page, not a live fetch. It shows the raw HTML, and under More info, the HTTP response headers, page resources and JavaScript console output. In the Coverage section, note the Last crawl date and the Crawled as value (mobile or desktop).
  2. Run Test live URL and open View tested page. This fetches the page now. If the test succeeds, it also shows a screenshot, which the indexed view never has.
  3. Compare the two. If the crawled HTML is empty or a challenge page and the live test shows your content, the problem was intermittent or has been fixed. If both are empty, it is happening now and you can reproduce it.

One limit matters here. The live test fetches as Google-InspectionTool, not as Googlebot. Google's list of common crawlers gives its user agent as Mozilla/5.0 (compatible; Google-InspectionTool/1.0) on desktop, and the mobile string also names Google-InspectionTool rather than Googlebot. A WAF or CDN rule that matches the word "Googlebot" can treat the live test differently from the real crawler, so a clean live test does not clear the edge. The URL Inspection tool guide covers the rest of the tool in detail.

Check the raw response: status, type and size

Next, look at the response without a browser. Request the page with Googlebot's smartphone user agent and save headers and body separately. Google publishes the string with a W.X.Y.Z placeholder for the Chrome version, which changes over time; any current version number works for this test.

bash
UA="Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

curl -s -A "$UA" --compressed -D headers.txt -o body.html \
  -w "status=%{http_code} type=%{content_type} downloaded=%{size_download}\n" \
  https://www.example.com/page

wc -c < body.html                  # uncompressed size in bytes
grep -o "<p[ >]" body.html | wc -l # rough count of paragraph tags

With --compressed, size_download is the transferred (compressed) size, while body.html is saved decoded, so wc -c gives the uncompressed size that Google's limit applies to.

Then read the output against what Google documents:

  • Status. Google's HTTP status code documentation says that for a 200, Google passes on whatever it received to the next step, which for Search is the indexing pipeline. An empty 200 is therefore passed on as the page, although Google also says Search Console may report an empty page as a soft 404. A 204 No Content means Google received no content and cannot process it.
  • Content-Type. Google's indexable file types page says the file type is determined by the Content-Type header Google gets when it crawls the file. If that header is missing or incorrect, for example an HTML page sent as application/octet-stream, Google may use the file extension or re-parse the file with a different parser, and the result may not be what you expect. The Content-Type and encoding guide covers charset and compression mistakes on that layer.
  • Size. The Googlebot documentation says Googlebot crawls the first 2 MB of a supported file type, measured uncompressed, and sends only the downloaded part for indexing consideration. If inline data fills the first 2 MB, the text may sit past the cutoff. See the Googlebot file size limit for how to measure it.
  • Body. A paragraph count of zero, a body under a few kilobytes, or text like "checking your browser" suggest the content is not in the response.

Run the same request with a normal browser user agent. If the two bodies differ, the difference comes from something that reads the user agent, which points to the edge or to bot-detection code in your application.

When Googlebot gets a different response than you do

A curl test only shows rules keyed on the user agent. Bot rules can also look at the IP address, the TLS fingerprint or a reputation score, so real Googlebot can get a block page that your test never triggers. Check the logs for requests that really came from Google.

Google's Googlebot verification guide gives two methods. One is a reverse DNS lookup on the IP, a check that the name ends in googlebot.com, google.com or googleusercontent.com, and a forward lookup to confirm it resolves back:

bash
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The other is to match the IP against Google's published ranges, for example common-crawlers.json for Googlebot. Start with log lines that claim to be Googlebot, then keep only the IPs that pass one of those checks:

bash
grep "Googlebot" access.log | awk '{print $1, $9, $10, $7}' | sort | uniq -c | sort -rn | head

In the common and combined log formats, $1 is the client IP, $7 the path, $9 the status and $10 the response bytes. A run of 200 responses with a tiny, identical byte count for pages that are normally large is the signature of an edge or origin serving something else. Then open the WAF or CDN event log for the same timestamps. The WAF challenge page guide shows what common challenge responses look like and how to allow verified crawlers safely.

Also check the caching layer. If a CDN once cached an empty or error body under a 200 status, every crawler that gets that cached object receives the same empty page until it expires or is purged.

Is it cloaking?

Google's spam policies define cloaking as presenting different content to users and search engines with the intent to manipulate rankings and mislead users. An accidental challenge page is not that, but the fix is the same either way: Googlebot should get the same content a person gets. If you deliberately vary content by user agent, read serving crawlers different content before going further. Google also notes that when a site is compromised, it is not uncommon for the attacker to use cloaking so the owner is less likely to notice, so an unexpected difference you did not build deserves a security check.

Empty 200 responses from an origin under load

A cause that leaves no trace in a later test is an origin that answers 200 with an empty or partial body when it is overloaded: a timed-out database call caught by a template that renders nothing, a worker that runs out of memory after sending headers, or a fallback page with no text. When you test the next day, everything looks fine.

Google's documentation points to a better failure mode. Its guide to reducing crawl rate says to return 500, 503 or 429 instead of 200 when you need Google to slow down, and warns against doing so for longer than one or two days, because a URL that keeps returning these codes may be dropped from the index. Google's status code page adds that it ignores any content it receives with a 5xx status, and that already indexed URLs are kept in the index for a while but eventually dropped if the errors continue. A short burst of 503s therefore leaves the indexed copy alone, while an empty 200 is passed on to indexing as the page.

http
HTTP/1.1 503 Service Unavailable
Content-Type: text/html; charset=utf-8

To fix it, make the application fail loudly. If a required data call fails, return a 5xx rather than a page shell. If you have a monitor, alert on 200 responses whose body is below a size you would never expect for that template.

How to fix it and confirm the fix

Work through the layers in order and stop at the one that explains the evidence:

  1. Origin: pages return the full HTML with a 200, or a 5xx when they cannot.
  2. Headers: Content-Type: text/html; charset=utf-8 for HTML pages, and no 204 on real pages.
  3. Size: the main text and links sit well inside the first 2 MB of uncompressed HTML.
  4. Edge: verified Googlebot IPs are not challenged or blocked, and no cached empty body is being served.
  5. Rendering: if the text only appears after JavaScript runs, make sure the scripts and API calls are not blocked for Google.

After the fix, run a live test, confirm the tested page shows your content, and request indexing for a few important URLs. Then use Validate fix in the Page indexing report. Google says validation typically takes up to about two weeks and in some cases much longer. You can also skip validation: Google updates the issue count whenever it crawls a page with a known issue.

Frequently asked questions

Does page indexed without content mean my page is blocked?

No. Google says this status is not a robots.txt block, and the page is in the index. The problem is that the content Google received could not be read.

Why does the live test look fine when the status persists?

The status reflects the last crawl, while the live test fetches now and as Google-InspectionTool rather than Googlebot. An intermittent origin failure or an edge rule that targets Googlebot can produce an empty crawl that a later live test does not reproduce.

Can a WAF challenge cause this status?

Google does not name WAFs as a cause, but a challenge or block page served with a 200 status is passed on to indexing in place of your content, which fits Google's description. Check your WAF events for verified Googlebot IPs around the last crawl date.

How long does it take to clear after a fix?

Google says validation typically takes up to about two weeks and in some cases much longer. Google also updates the report when it recrawls affected pages, even without a validation request.

Does this status affect AI search tools too?

The status is Google's, but the causes are not. An empty 200, a challenge page or a wrong Content-Type is served to any crawler that triggers it, including AI crawlers, and Search Console will not tell you about those.

See what crawlers receive from your page

The quickest way to rule out the edge and the raw response is to fetch the page the way a crawler does. Run the free check on an affected page to see whether the main AI crawlers can fetch it. It covers a single page and needs no account.

Filed under Rendering and content. Topics: search-console, google, rendering, waf, troubleshooting