Skip to content

Storebot-Google: Google's shopping crawler and robots.txt

What Storebot-Google is, its two user agent strings, why it ignores your Googlebot group in robots.txt, and how to verify it and keep it out of cart traps.

By Crawlito · 7 min read

In short

  • Storebot-Google is one of Google's common crawlers, and Google says rules addressed to it affect all surfaces of Google Shopping, such as the Shopping tab in Google Search.
  • Its robots.txt token is Storebot-Google, and Google's own robots.txt spec says it follows the * group, not the Googlebot group, when no group names it.
  • Google says the crawler visits product detail, cart and checkout pages and moves through checkout without buying anything, recording data such as prices, shipping, availability and payment methods.
  • Google says its common crawlers generally crawl from the common-crawlers.json IP ranges and reverse-resolve to crawl----.googlebot.com or geo-crawl----.geo.googlebot.com hostnames.
On this page

Storebot-Google is Google's shopping crawler. Google lists it among its common crawlers, gives it the robots.txt token Storebot-Google, and says that crawling preferences addressed to it affect all surfaces of Google Shopping. It sends a desktop and a mobile user agent string, obeys robots.txt, and does not read your User-agent: Googlebot group: if no group names it, it follows User-agent: *.

This post is a reference for the crawler itself: the strings, the robots.txt matching, how to verify a request, and the store-specific URLs it is likely to touch. If Merchant Center is reporting that it cannot check your product pages, the fixes for those errors are in Merchant Center unable to check product pages.

What Storebot-Google is and what it crawls

Google's list of common crawlers names it "Google StoreBot" and says rules for the Storebot-Google token affect Google Shopping surfaces, for example the Shopping tab in Google Search and Google Shopping. Like the other common crawlers, it always obeys robots.txt rules when crawling automatically.

Google's Merchant Center help page on Google StoreBot describes what it does on a store:

  • It crawls product detail pages, cart pages and checkout pages.
  • It tries to move through the checkout flow, using machine learning to fill in forms such as shipping addresses, and records data such as product and shipping prices, availability, coupon validity and payment methods, without making a purchase.
  • Google says the data is used to improve the user experience and to help verify information you submit through Merchant Center.
  • Structured data and Merchant Center settings take precedence over what the crawler extracts.

Google "highly encourages" allowing it, and for sites without a Merchant Center account that still want to block it, the page points to Google's general crawling documentation. The page does not say how often it visits, and Google does not document whether its data is used for AI Mode or AI Overviews. Treat it as a Shopping crawler, not an AI search crawler.

One thing that surprises people: Google's Merchant Center help page on product crawl issues asks you to allow Googlebot (landing pages) and Googlebot-image (images). It does not mention Storebot-Google. They are separate crawlers with separate tokens, so a rule that suits one does not automatically apply to the other.

The two Storebot-Google user agent strings

Google documents one desktop and one mobile string. W.X.Y.Z is a placeholder for the Chrome version, which changes over time.

Desktop:
Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36

Mobile:
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36

Both strings contain Storebot-Google/1.0, so that is the substring to match in log filters and WAF rules. Do not match on the Chrome version or the Android device model. Neither string contains Googlebot, which means a rule that only matches Googlebot will miss it, and a rule that matches Google will catch it along with many other agents. The Merchant Center help page writes the name as StoreBot-Google. Google matches the robots.txt user-agent value without regard to case, but a case-sensitive log filter or WAF rule would treat the two spellings as different, so match without case.

The same pattern for Googlebot's own strings is covered in Googlebot user agent strings.

bash
# Count Storebot-Google requests per URL in an access log
grep -i 'Storebot-Google/1.0' access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

How robots.txt rules apply to Storebot-Google

Google's robots.txt specification says only one group is valid for a crawler: the group with the most specific user agent that matches it. Other groups are ignored, and a named group is not combined with the * group.

Google's own precedence example uses Storebot as a case. With groups for googlebot-news, * and googlebot, the spec says Storebot-Google follows the * group, because there is no group for Storebot-Google. The Googlebot group does not count.

Your robots.txt hasStorebot-Google follows
Only User-agent: *The * group
User-agent: * and User-agent: GooglebotThe * group
A User-agent: Storebot-Google groupThat group only, and the * group is ignored
No groups that match and no * groupNo rules, so everything is allowed (RFC 9309)

That has two practical consequences.

A Googlebot-only allowance does not reach it. If your * group blocks a path and you opened it for Googlebot in its own group, Storebot-Google is still blocked from it.

A Storebot-Google group replaces your general rules. If you add a named group just to allow it, every Disallow you wrote under * stops applying to it. Repeat the rules you still want inside the named group. The post on how robots.txt groups work walks through this accident in detail.

A store file that keeps Storebot-Google on the same footing as everyone else, but names it explicitly, looks like this:

User-agent: *
Disallow: /account/
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*filter=

User-agent: Storebot-Google
Disallow: /account/
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*filter=

If you have no reason to treat it differently, the simplest option is to leave it out of the file and let it follow *.

Cart, checkout and faceted URLs: the store-specific risk

Storebot-Google is unusual because Google says it deliberately visits cart and checkout pages. Most store robots.txt files block those paths for everyone, and that * rule applies to Storebot-Google too. That is a choice to make on purpose, not by accident:

  • If your * group disallows /cart or /checkout, Storebot-Google will not crawl them. Google's help page does not describe what happens to Shopping data in that case, so it is not documented whether blocking them has any effect. It does say structured data and Merchant Center settings take precedence over what the crawler extracts, which is a reason to keep product structured data and Merchant Center shipping and price settings complete.
  • If you allow those paths, expect cart and checkout requests in your server logs that come from a crawler, not a shopper. Google says it does not buy anything, but your own systems may still count the visits. Check that bot traffic is excluded from cart and abandoned-checkout reporting.
  • Add-to-cart links as plain URLs are a trap. Links such as /cart/add?id=123&qty=1 or ?add-to-cart=123 change state when fetched with GET. Any crawler that follows them can fill carts, create sessions and generate endless URL variants. Use POST forms or buttons for cart actions, and disallow the GET endpoints if they exist.

Faceted navigation is the other common problem. Filter and sort parameters produce many combinations of the same product list, and a shopping crawler that respects your rules will still fetch whatever you leave open. Decide which filtered pages deserve to be crawled and block the rest, as described in faceted navigation crawl traps. Product detail pages themselves should never be caught by those patterns: test your Disallow lines against a real product URL, including variant parameters, before you publish them.

# Risky: this also blocks /products/shirt?variant=123
Disallow: /*?

# Narrower: blocks only the facet parameters
Disallow: /*?*sort=
Disallow: /*?*filter=

How to verify a real Storebot-Google request

Any client can put the Storebot-Google string in its User-Agent header, so the string alone proves nothing. Because Storebot-Google is a common crawler, Google's verification guide applies to it in the same way as to Googlebot:

  1. Run a reverse DNS lookup on the IP from your logs with host.
  2. Check that the hostname is in googlebot.com. For common crawlers Google gives the masks crawl-***-***-***-***.googlebot.com and geo-crawl-***-***-***-***.geo.googlebot.com.
  3. Run a forward lookup on that hostname.
  4. Check that it returns the original IP.
bash
# Google's own example from its verification guide
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For large-scale matching, use the IP ranges in Google's common-crawlers.json, which Google says the common crawlers generally use. A request with the Storebot-Google string that fails the reverse and forward DNS check is not from Google, whatever its user agent says. The general method, including other operators, is in how to verify real AI crawlers.

If your WAF allowlists Googlebot by user agent, check that the rule also covers Storebot-Google, or better, that it allows verified Google crawlers by IP range and reverse DNS rather than by string.

Frequently asked questions

Does Storebot-Google follow my Googlebot rules in robots.txt?

No. Google's robots.txt specification says Storebot-Google follows the * group when there is no group that names Storebot-Google, even if a Googlebot group exists.

Is Storebot-Google an AI crawler?

Google documents it as a common crawler whose rules affect Google Shopping surfaces. Google does not document that its data is used for AI Mode or AI Overviews.

Should I block Storebot-Google?

Google's help page says allowing it is highly encouraged, and Google's crawler documentation says rules addressed to Storebot-Google affect all surfaces of Google Shopping. If you do block it, do it with a named group in robots.txt so the decision is visible.

Why does Storebot-Google visit my cart and checkout?

Google says it moves through the checkout flow to record prices, shipping, availability, coupon validity and payment methods, without buying anything. It will only do so on paths your robots.txt allows it to crawl.

How can I tell a fake Storebot-Google from a real one?

Run a reverse DNS lookup on the IP and check for a googlebot.com hostname, then confirm with a forward lookup, or match the IP against Google's common-crawlers.json ranges.

Check which crawlers can reach your product pages

Storebot-Google is one of several crawlers a store has to account for in robots.txt and at the firewall. Run the free check on one of your product pages to see whether the main search and AI crawlers can fetch it, or book an audit if you want robots.txt groups, WAF rules and faceted URLs reviewed across the store with evidence.

Filed under AI crawlers. Topics: google, ecommerce, robots-txt, ai-crawlers