Skip to content

Server error (5xx) in Search Console: find what Googlebot hit

Why Search Console reports Server error (5xx) when your site loads fine, how Google treats 5xx and 429, and how to find the failing requests in Crawl stats and logs.

By Crawlito · 9 min read

In short

  • Search Console's Server error (5xx) means Googlebot got a 500-level response, timed out, or found the site busy when it requested the URL, not that the page fails for you now.
  • Google treats 429 as a server error too: both 5xx and 429 make its crawlers slow down, and URLs that keep returning them are eventually dropped from the index.
  • The Crawl stats report groups Googlebot's requests by response, including Server error (5XX) and Page timeout, with example URLs and a host status for the past 90 days.
  • Googlebot sends requests without Accept-Language, its default IP addresses appear to be in the US, and it clears cookies between page loads, so code paths that only run for crawlers can fail unseen.
  • Confirm recovery with a URL Inspection live test, clean 200s to verified Googlebot in your logs, then Validate fix, which Google says typically takes up to about two weeks.
On this page

Server error (5xx) in Search Console's Page indexing report means that when Googlebot requested the URL, your server returned a 500-level error, the request timed out, or the site was busy. It describes one past request, not what you see in a browser today. To find the cause, look at the requests Googlebot actually made: the Crawl stats report shows them grouped by response, and your access logs, filtered to verified Googlebot, show the exact time, status and path of each failure.

The usual reasons a site looks fine to you but fails for Googlebot are timing (errors under load or at night), layers (a CDN or proxy error rather than the origin) and request shape (Googlebot sends no Accept-Language header and does not keep cookies between page loads, so a different code path runs). This guide works through each one and ends with how to confirm the fix.

What Server error (5xx) means in Page indexing

Google's Page indexing report help defines the issue as "Your server returned a 500-level error when the page was requested", and lists three situations behind it: Googlebot could not access the URL, the request timed out, or the site was busy. Its advice is to check host status in the Crawl stats report, make sure the hosting server is not down, overloaded or misconfigured, and look for a firewall or protection system that may be blocking Googlebot.

Two consequences follow. First, a timeout counts, so a page that eventually loads in your browser after many seconds can still be reported. Second, a firewall rule that answers Googlebot with a 5xx error page shows up here too, not as a block.

How Google treats 5xx and 429 responses

Google's page on HTTP status codes and network errors documents the behaviour:

ResponseWhat Google's crawlers do
500, 502, 503Decrease the crawl rate for the site, in proportion to the number of URLs returning a server error; content from those responses is ignored
429Treat it as a signal that the server is overloaded; it is considered a server error
Persistent 5xx or 429Already indexed URLs are kept for a while, but eventually dropped
Back to 2xxGoogle gradually increases the crawl rate again
Other 4xx, such as 403No effect on crawl rate; the content is treated as nonexistent

Google's crawling troubleshooting guide adds a time limit: returning 503 or 429 for more than two days will cause Google to drop those URLs from the index. If the 5xx responses are deliberate, for maintenance, the 503 maintenance mode guide covers how to do it safely. This post is about the 5xx errors you did not intend.

Step 1: read Crawl stats by response and host status

The Crawl stats report is under Settings (property settings) in Search Console, then Crawl stats, and is only available for root-level properties: a Domain property or a root URL-prefix property. It is the closest thing to Googlebot's own log of your site.

Start with Host status. It covers the past 90 days and checks three things separately: robots.txt fetching, DNS resolution and server connectivity. Server connectivity flags times when your server was unresponsive or did not give a full response. Each chart has a red line, and days above it count as an issue. A server connectivity problem points at capacity or network trouble across the host rather than a single broken template.

Then open By response in the crawl requests table. Google lists these responses under bad:

  • Server error (5XX)
  • Page timeout
  • Page could not be reached
  • Fetch error
  • DNS unresponsive and DNS error

Click Server error (5XX) to see example URLs, then click a URL for the details of that request. Note the dates and the paths. Three patterns are worth looking for:

  1. One template. All examples are product pages, or all are search results. That points at application code or a slow query on that template.
  2. One time window. Errors cluster at the same hours. That points at backups, cron jobs, cache purges or deploys.
  3. One host. With a Domain property you can compare subdomains. Errors on cdn. or api. alone narrow it down quickly.

Google says the example URLs are representative, not complete, and that some requests may not be counted, so the report gives you leads, not totals. Your logs give you totals.

Step 2: find the failing requests in your access logs

Filter your access log to Googlebot and to 5xx statuses. With the common combined log format:

bash
grep "Googlebot" /var/log/nginx/access.log \
  | awk '$9 ~ /^5/ {print substr($4, 2, 14), $9, $7}' \
  | sort | uniq -c | sort -rn | head -30

Each output line is a count, the hour (such as 15/Nov/2026:03), the status and the path, so clusters by time and by path stand out. The $4, $9 and $7 columns are the timestamp, status and path in the combined format; adjust them if yours differs. If the log has a request time field, add it, because timeouts often show as long requests followed by a 504 from a proxy. The server logs guide explains log locations and formats.

Verify that it is really Googlebot

Plenty of scrapers use the Googlebot user agent, and their errors do not matter to Search Console. Google's verification page describes a reverse DNS lookup on the IP, a check that the name ends in googlebot.com, google.com or googleusercontent.com, and a forward lookup that must return the same IP:

bash
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For bulk checks, Google publishes its IP ranges as JSON, starting with common-crawlers.json for Googlebot. Match the failing IPs against it and discard the rest.

Compare the error times with the Crawl stats dates

If verified Googlebot requests show 5xx at the same times as the Crawl stats examples, you have found the failures. If your origin log shows only 200s for those requests, the error came from a layer in front of it, which is the next step.

Step 3: check whether the CDN or the origin returned the error

Some 5xx errors Googlebot sees are never written to the origin log, because a CDN, load balancer or reverse proxy generated them. A proxy that cannot reach the origin, or gives up waiting, returns its own 502, 503 or 504 page.

To tell them apart:

  • Look at the error response itself. When you can trigger the error, fetch it with curl -i and read the headers and body. A server header naming your CDN and a branded error body point at the edge. URL Inspection will not show this: Google's URL Inspection help says the HTTP response details under View crawled page and More info are only available when the page was fetched successfully.
  • Compare the logs. An error in the CDN log with no matching origin request means the edge failed to connect or timed out. A matching origin request with a 5xx means the application failed.
  • Check bot rules that return 5xx. Some firewalls and hosting protections answer suspected bots with a 503 instead of a 403, which then appears as a server error.

The Cloudflare codes have their own guide: Cloudflare 520, 521, 522 and 524 errors. On shared hosting, also check for a hosting resource limit, which some hosts report as a 508 or 503.

Step 4: test for errors that only happen to crawlers

When the logs confirm that verified Googlebot got 5xx but you cannot reproduce it, the request itself is usually different from yours.

Concurrency and timeouts

You test one page at a time. Googlebot uses parallel connections, and Google's crawl budget guide says its crawl capacity limit factors in both the number of those connections and their duration. During a burst, workers run out, database connections queue and requests time out. Look at whether the errors cluster at the peaks of crawl requests in Crawl stats, or at the peaks of all bot traffic in your logs, since AI crawlers hitting the same server add load. For measuring and fixing slow responses, see slow TTFB and crawler timeouts.

You can reproduce a small burst on one URL. The changing query string usually bypasses a CDN cache, so the requests reach your origin:

bash
seq 1 20 | xargs -P 10 -I{} curl -s -o /dev/null \
  -w "%{http_code} %{time_total}\n" \
  "https://www.example.com/products/blue-widget?nocache={}"

Keep it small and run it at a quiet time. A few 502s or 504s here, where a single request succeeds, confirm a capacity problem.

No cookies and no Accept-Language

Google documents that Googlebot sends HTTP requests without an Accept-Language header and that its default IP addresses appear to be in the USA (locale-adaptive pages). Its JavaScript guide says the renderer clears cookies, local storage and session storage across page loads (fix search-related JavaScript problems).

That matters when code assumes those values exist. Typical failures are a locale lookup that throws when the header is missing, a session or consent middleware that errors on a first visit, or a geo redirect that loops for US addresses. Reproduce the request the way Googlebot sends it, with no cookie jar and no language header:

bash
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -H "Accept-Language:" \
  https://www.example.com/products/blue-widget

curl sends no Accept-Language header and no cookies by default; the empty -H "Accept-Language:" makes that explicit. If this returns 500 while your browser gets 200, check the application error log for that timestamp. The test does not copy Googlebot's IP addresses, so rules keyed on IP or country can still behave differently for the real crawler. Language and location handling has its own guide in Accept-Language redirects and crawlers.

Step 5: confirm the fix and validate

Once you have changed something, check it in order:

  1. Live test. In URL Inspection, run Test live URL on several affected URLs. Google notes that server errors can be transient, so a pass on one URL is a good sign, not proof.
  2. Logs. Watch verified Googlebot requests to the affected paths for a few days. You want 200s at the times the errors used to happen.
  3. Crawl stats. The share of Server error (5XX) responses should fall, and host status should stop flagging new server connectivity issues.
  4. Validate fix. On the issue's page in the Page indexing report, click Validate fix. Google says validation typically takes up to about two weeks and sometimes much longer. Do not start it again until it has passed or failed; if it fails, the button becomes Start new validation.

If validation fails, the next steps are in Search Console validation failed.

Frequently asked questions

Why does Search Console show Server error (5xx) when my page loads fine?

The report records what happened when Googlebot fetched the URL, which may have been days ago, under load, or through a different path than your browser. Check Crawl stats by response and your logs for the date of that request.

Does a 429 response count as a server error?

Yes, for crawling. Google says its crawlers treat 429 as a sign that the server is overloaded and consider it a server error, so it slows crawling the same way 5xx does.

How long can URLs return 5xx before Google drops them?

Google says returning 503 or 429 for more than two days will cause it to drop those URLs from the index. Other persistent server errors also lead to removal, without a published number of days.

Should I return 403 instead of 5xx to stop Googlebot overloading my server?

No. Google says 4xx codes other than 429 have no effect on crawl rate and that it treats the content as nonexistent, so a 403 removes pages instead of slowing the crawl.

Check what crawlers get from your pages

Server errors are one of several ways a crawler can fail to get a page you think is live. Run the free check on an affected URL to see the status and content a crawler receives, or ask for an audit if you want status codes, edge settings and crawler access reviewed across the site with evidence.

Filed under Access and robots.txt. Topics: http-status, search-console, google, server-logs, troubleshooting