Skip to content

Google-InspectionTool: the crawler behind URL Inspection tests

Google-InspectionTool fetches pages for URL Inspection live tests and the Rich Results Test. Learn its user agents, robots.txt token and how to allow it safely.

By Crawlito · 8 min read

In short

  • Google-InspectionTool is the crawler that Google's Search testing tools use, including the Rich Results Test and the URL Inspection live test in Search Console.
  • Its desktop and mobile user agent strings contain Google-InspectionTool/1.0 and do not contain the word Googlebot, so a firewall rule that allows only Googlebot by user agent will not let live tests through.
  • In robots.txt it answers to two tokens, Google-InspectionTool and Googlebot, and Google says rules aimed at Google-InspectionTool affect only the testing tools, not Google Search.
  • Google lists it among its common crawlers, which generally crawl from the IP ranges in common-crawlers.json with hostnames under googlebot.com, so you can verify it the same way as Googlebot.
  • A live test that fails while Googlebot crawls normally often points to a rule that treats the two user agents differently, so compare the status codes your logs show for each before changing the page.
On this page

Google-InspectionTool is the crawler Google uses for its Search testing tools: the URL Inspection live test in Search Console and the Rich Results Test. It sends its own user agent strings, which do not contain the word "Googlebot", and it has its own robots.txt token. If your live tests fail while Googlebot crawls your site without trouble, a common cause is a firewall, plugin or robots.txt rule that recognises Googlebot but not Google-InspectionTool.

This post covers the strings, how robots.txt applies to it, why "Googlebot only" rules break the tools, how to read the failures, and how to allow it without allowing impostors. For how to use the URL Inspection tool itself, see the URL Inspection guide.

What Google-InspectionTool is and what it fetches

Google lists Google-InspectionTool on its common crawlers page. It says that crawling preferences addressed to this user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console, and that they have no effect on Google Search or other products.

Two Google help pages confirm which tools use it:

  • The URL Inspection help page says the live URL test only confirms whether Google-InspectionTool can access your page for indexing, and that the screenshot shows how Google-InspectionTool sees the page.
  • The Rich Results Test help page says the tool accesses the page as Google-InspectionTool, that you can choose a smartphone or desktop user agent, and that smartphone is the default.

Google documents the crawler only in connection with these testing tools, so a genuine request in your logs most likely means someone, possibly you, a colleague or an agency, ran a test on that URL. It is not a Googlebot crawl, and the live test does not add what it fetches to Google's index.

The desktop and mobile user agent strings

Google publishes two strings:

Desktop:
Mozilla/5.0 (compatible; Google-InspectionTool/1.0)

Mobile:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)

Google says Chrome/W.X.Y.Z is a placeholder for the Chrome version, which increases over time, and that you should use a wildcard for the version when you search logs or filter traffic.

The difference that matters is in the last part. Googlebot Smartphone uses the same Nexus 5X prefix but ends in compatible; Googlebot/2.1; +http://www.google.com/bot.html. Google-InspectionTool ends in compatible; Google-InspectionTool/1.0. Neither of its strings contains Googlebot. The full list of Googlebot strings, and what else shares the Nexus 5X prefix, is in Googlebot user agent strings.

To find it in your logs:

bash
grep -i "Google-InspectionTool" access.log | tail -20

Check the status code on each line. A 200 means the test got the page. A 403, 429 or 5xx means something on your side refused or failed the test request.

How robots.txt applies to Google-InspectionTool

Google lists two robots.txt tokens for this crawler: Google-InspectionTool and Googlebot. The same page says you need to match only one token for a rule to apply. Combined with Google's robots.txt specification, which says a crawler follows only the group with the most specific user agent that matches it and ignores the other groups, this gives three cases. Google does not spell these out for Google-InspectionTool itself; they follow from the spec's own example, where googlebot-news follows its own group rather than a googlebot group.

Your robots.txt hasGoogle-InspectionTool follows
A Google-InspectionTool groupThat group only
No Google-InspectionTool group, but a Googlebot groupThe Googlebot group
NeitherThe * group

The first row has a catch. Once you add a Google-InspectionTool group, the tool stops following your Googlebot rules, because Google uses only the most specific matching group and ignores the rest. That can produce two misleading results:

User-agent: Googlebot
Disallow: /checkout/

User-agent: Google-InspectionTool
Allow: /

Here the live test reports /checkout/ as crawlable while Googlebot is blocked from it. The reverse also happens: a Disallow in a Google-InspectionTool group makes the live test report "Crawl allowed? No" while Googlebot crawls the page normally.

If you want the tests to reflect what Googlebot gets, do not give Google-InspectionTool a group of its own. It will then follow your Googlebot group. robots.txt groups explained covers group matching in more detail.

Blocking the tool in robots.txt does not remove anything from Google Search, according to Google. It only stops you, and anyone else, from testing those URLs.

Why "allow Googlebot" firewall rules break live tests

Many sites protect themselves with a rule that challenges or blocks bots and exempts "good" crawlers by user agent. If the exemption checks for googlebot, it covers Googlebot and its image and video variants, but not Google-InspectionTool. A simplified nginx version shows the problem:

nginx
map $http_user_agent $is_search_bot {
    default        0;
    ~*googlebot    1;
}

# Later, a bot check that blocks non-browsers unless $is_search_bot is 1

Googlebot crawls fine, the pages stay indexed, and every URL Inspection live test or Rich Results Test fails, because Google-InspectionTool/1.0 never matches googlebot. Security plugins and WAF allowlists that use a list of known crawler names can have the same gap if the list predates the tool or was copied from an old Googlebot-only example.

The opposite situation also occurs. Rules that block by request pattern, such as rate limits or "headless browser" detection, may catch the tool on one test and let it through on the next. A passing test is not proof that Googlebot gets through, and a failing test is not proof that Googlebot is blocked. When the two disagree, compare the status codes your logs show for each user agent.

For the WordPress-specific version of this problem, see security plugins that block crawlers.

How to read the common live-test failures

The URL Inspection help page splits live-test failures into page-level and site-wide ones.

Page level. The "Page fetch" field shows Successful, Failed with a status reason, Internal error or N/A. Google says that if crawling is not allowed by robots.txt, this field shows a failure, and that a fetch can still fail when crawling is allowed. The status reasons are the same ones used in the page indexing report, such as 403, 404 or 5xx. If you see "Blocked due to access forbidden (403)" on a live test, the Googlebot 403 guide walks through finding the layer that refused it. The difference is that you search your firewall and logs for Google-InspectionTool as well as Googlebot.

Site wide. When the availability status shows "Not available due to a site-wide issue", Google lists these Page fetch values:

Page fetch valueWhat Google saysWhat to check
DNS server unresponsiveOften transient, resolves in a few minutesRetry, then check your DNS provider
DNS error: Host unknownYour DNS did not resolve the URLThe hostname's DNS records
DNS error: Private IP providedDNS returned a private range IP such as 10.0.0.1Records pointing at internal addresses
Server connection errorServer unreachable, refused or failed to connectFirewall rules that drop Google's IPs, origin uptime
Server invalid responseUnsupported protocol or bad responseProxy or compression misconfiguration
Invalid server SSL certificateGoogle will not test an HTTPS URL without a valid certificateCertificate chain and expiry
Robots.txt unreachableGoogle will not crawl if robots.txt exists but cannot be reachedWhether /robots.txt returns 5xx or times out
Hostload exceededYour site appears at capacity for crawling or inspection requestsServer load; wait and retry

If a "Server connection error" appears only on live tests, one possible cause is a network-level rule that drops the tool's requests before they reach the web server, in which case your access log shows nothing at all.

Google also notes that server errors can be transient: a live test can hit an error that crawling does not, and the other way round. Run the test twice before changing anything.

The Rich Results Test reports the same kind of problems in its own wording. Its help page mentions errors such as "DNS server unresponsive" and "Server connection error", and says all page resources must be accessible to an anonymous user from the internet. If your rich results test passes but the results still do not appear, that is a different problem, covered in rich results not showing despite valid markup.

How to verify and allow Google-InspectionTool

Do not allow the user agent string on its own. Anyone can send it, and Google's crawler page warns that user agent strings can be spoofed.

Google lists Google-InspectionTool on its common crawlers page, and says common crawlers generally crawl from the IP ranges in common-crawlers.json with reverse DNS hostnames that match crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Google's verification guide gives two methods:

  1. Reverse and forward DNS for one-off checks. Take the IP from a log line with Google-InspectionTool, look up its hostname, confirm it ends in googlebot.com, google.com or googleusercontent.com, then look up that hostname and confirm it returns the same IP.

    bash
    host 66.249.66.1
    host crawl-66-249-66-1.googlebot.com
  2. Published IP ranges for automated rules. Match the IP against common-crawlers.json, and refresh the file on a schedule rather than copying the ranges once.

Then change the rule that blocked the tests:

  • Allowlists by name: add Google-InspectionTool next to Googlebot, and pair both with the IP or DNS check.
  • nginx or Apache maps: extend the pattern, for example ~*(googlebot|google-inspectiontool), and keep the IP verification step that guards it.
  • WAF or CDN rules: if the exemption uses a verified-bot field, check your provider's documentation to see whether it covers Google-InspectionTool. If it is not covered, add an exemption for the token restricted to Google's published ranges.
  • robots.txt: remove any Google-InspectionTool group you did not add on purpose, so the tool follows your Googlebot rules.

After the change, rerun Test live URL on an affected page and confirm the log line now shows a 200.

For a page behind a firewall or on a local machine that you cannot open up, Google's help pages say you can expose it to the tools through a tunnel.

Frequently asked questions

Does blocking Google-InspectionTool affect my rankings?

Google says crawling preferences addressed to Google-InspectionTool affect only Search testing tools and have no effect on Google Search or other products. Blocking it stops the tests from working, and it does not tell Googlebot anything.

Why is Google-InspectionTool in my logs when I did not run a test?

Google documents it for its Search testing tools, so a genuine request usually means someone ran a URL Inspection live test or a Rich Results Test on the URL. The Rich Results Test is not limited to Search Console users of your property, so the request may come from anyone testing your page, and a spoofed user agent is also possible until you verify the IP.

Does a passing live test mean Googlebot can crawl the page?

Not necessarily. The test uses a different user agent, and Google says live test results can differ from crawling because of transient server errors. Confirm with your logs that verified Googlebot requests get a 200.

Is Google-InspectionTool used by AI Overviews or Gemini?

Google does not say so. Its documentation ties the crawler to Search testing tools only. Google-Extended is the token Google documents for Gemini training and grounding choices.

Check what crawlers receive from your pages

Fixing a user agent rule for one Google tool is a good moment to check the others. The free check fetches one page the way the main AI crawlers do and shows their robots.txt rules and what is in the raw HTML, with no account. It runs from Crawlito's servers, not Google's, so Search Console and your logs stay the evidence for Google-InspectionTool itself.

Filed under AI crawlers. Topics: google, search-console, waf, troubleshooting