Skip to content

Unable to check product pages in Merchant Center: robots.txt fix

Why Merchant Center says it is unable to check product pages, how to find the robots.txt, firewall or HTTP error behind it, and how long re-crawling takes.

By Crawlito · 8 min read

In short

  • Google says Unable to check product pages means it cannot crawl the landing page in the link attribute, because robots.txt blocks its crawler or the page returns an HTTP error.
  • Affected products stay disapproved until Google can crawl the landing page, and Google's fix is to let both Googlebot (landing pages) and Googlebot-image (images) crawl the site.
  • Google's product crawl issues page also lists 404s, 4xx and 5xx responses, pages that require a login, timeouts, too many redirects, firewalls, geo-restrictions and an unreachable robots.txt as causes.
  • Google suggests allowing crawling from the US, since it often crawls from there, and asking your host to let the Googlebot user agent through the firewall.
  • Google's pages give 24 to 72 hours for Needs attention to update and up to 48 hours for a fixed product to reappear in listings.
On this page

Merchant Center shows Unable to check product pages when Google cannot crawl the landing page you submitted in the link attribute. Google's help page for the issue names two causes: your robots.txt blocks Google's crawler, or the landing page returns an HTTP error when Google fetches it. The affected products stay disapproved and stop appearing in Shopping ads and free product listings until Google can crawl the page, so the fix is to find the block, remove it, and wait for the re-crawl.

You may also see the problem described as "page not crawlable due to robots.txt"; Google's current help page calls the issue Unable to check product pages. This post covers the crawl issues themselves. For a price disagreement between the feed and the page, see Merchant Center price mismatch after a sale. For Google's shopping crawler, which these help pages do not mention, see Storebot-Google.

The four Merchant Center crawl issues and what each means

Google spreads crawl problems across several help pages. They look alike in the Needs attention tab but point at different files:

Issue in Merchant CenterWhat Google could not fetchCrawler Google asks you to allow
Unable to check product pagesThe landing page in linkGooglebot and Googlebot-image
Product crawl issuesMobile and desktop landing pages and images, for many reasons beyond robots.txtGooglebot and Googlebot-image
Can't access product imagesImages in image_linkGooglebot and Googlebot-image
Robots.txt errorInformation about 3D assets on your siteGooglebot

The consequences differ. For landing pages, Google says a crawl error on either the mobile or the desktop page disapproves the item on both devices until it can access the page. For images, the help page says products are still shown for now but will soon be disapproved if robots.txt is not updated, because Merchant Center products require an image. The Robots.txt error page concerns 3D assets only and asks for a Googlebot group that allows the site.

Fix robots.txt for Googlebot and Googlebot-image

Google's example for the landing page and image issues is two groups with an empty Disallow, which allows everything:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

The image help page adds that this exact file is not required: any setup is fine as long as it does not block Google's crawlers. That matters, because pasting the example onto an existing file can change more than you intend.

Google's crawlers read only one group: the most specific one that matches them. Once you add a User-agent: Googlebot group, Googlebot stops reading your User-agent: * rules entirely. If your * group blocked /cart, /account or sort and filter URLs, those rules no longer apply to Googlebot. Repeat any you still want inside the new group, as explained in how robots.txt groups work.

Googlebot-Image has its own subtlety. If no group names it, it follows your Googlebot group, and only falls back to * when there is no Googlebot group. The details are in Googlebot-Image, Video and News tokens. A safe store file therefore keeps the private paths you need and makes sure product and image paths are open for both:

User-agent: *
Disallow: /account/
Disallow: /checkout/

User-agent: Googlebot
User-agent: Googlebot-image
Disallow: /account/
Disallow: /checkout/

Before publishing, check three patterns that commonly catch product URLs by accident:

  1. Broad query blocks such as Disallow: /*?, which also block /products/shirt?variant=123 when your feed links to variant URLs.
  2. Image or CDN paths such as /media/ or /cdn/ blocked to save crawl budget, which trigger Can't access product images.
  3. A separate image host. robots.txt is per host, so if image_link points at images.example.com, that host's own /robots.txt must allow Googlebot-image too.

Google's pages tell you to test with the URL Inspection tool in Search Console. Inspect one affected landing page URL exactly as it appears in the feed and read whether crawling is allowed.

Look past robots.txt: HTTP errors, firewalls and region blocks

Many stores have a clean robots.txt and still see the issue. Google's product crawl issues page lists the other causes it reports:

Cause Google listsWhat to check
Page not found (404)The link value is current and the product still exists
HTTP 4xx and 5xx responsesThe page returns a normal 200 to Google, not a 403, 429 or 503
Page requires authenticationNo login, password page or age gate in front of the product
Firewalls, geo-restrictions or private IPWAF, bot protection and country rules let Googlebot through
Timeout reading pageThe server answers quickly under crawler load
Server redirects too often, redirect errorsThe URL resolves in a short chain to the product
Hostname not resolvable, network errorDNS and TLS work from outside your network
Malformed HTTP responseA proxy, CDN or app server is not sending broken headers or bodies
Robots.txt unreachable/robots.txt itself returns 200 (or 404 if you have none), not a 5xx, a 429 or a timeout
Invalid URLNo invalid characters, and the link is a full valid URL
Regional landing pages not accessibleURLs with a region ID load for Google, if you use regional availability and pricing

On firewalls, Google's advice is specific: allow crawling from the US, because Google often crawls from there, and ask your webmaster or hosting provider to update firewall and server settings to allow the Googlebot user agent. A store that blocks every country but its own, or challenges unknown traffic with a JavaScript page, can pass every manual test from the office and still fail for Googlebot. Geo rules are covered in geo blocking crawlers, and the 403 case in Googlebot blocked with 403 Forbidden.

Allowing "the Googlebot user agent" should not mean trusting any request that claims it. Configure the rule to allow requests that pass Google's crawler verification, a reverse and forward DNS lookup or a match against Google's published IP range files, so a spoofed user agent does not get a free pass.

If you use the regional availability and pricing feature, the product crawl issues page says all landing pages with a region ID must be crawlable. URLs such as https://mystorewebsite.com/p/myproductlandingpage?region_id=123456 need to load for Google as well as the plain product URL, so check that no robots.txt rule or redirect strips or blocks the region_id parameter.

How to find the block in five checks

Take one affected product and work through it in order. Each check rules out a layer.

  1. Copy the exact URL. In Products, open the Needs attention tab, filter for the issue, open the product and copy the link value. For many products, download the .csv list from the same tab.
  2. Read robots.txt for that host. Open https://yourstore.com/robots.txt and find the group Googlebot follows. Check every Disallow against the URL path, including its query string.
  3. Request the URL with a Googlebot user agent. This does not prove what Google sees, but it shows whether your own stack treats the string differently.
bash
curl -sS -o /dev/null -w '%{http_code} %{url_effective} %{num_redirects}\n' -L \
  -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
  'https://yourstore.com/products/blue-shirt?variant=123'

A 403, 429 or 503, a long redirect chain, or a 200 that lands on a challenge or login page points at the server or WAF, not robots.txt.

  1. Check firewall and CDN events. Search your WAF or CDN logs for requests with Googlebot in the user agent to that path. A block at the edge never reaches your origin logs. If you have region rules, check whether US traffic is challenged or refused.
  2. Run URL Inspection. Use the live test for the same URL in Search Console; the URL has to be in a property you own. It reports Crawl allowed? and whether the Page fetch succeeded, and the tested page view shows the HTTP headers Google received.

Repeat the same steps for one image_link URL if the issue is Can't access product images.

How long the re-crawl takes after a fix

Google gives several timings, depending on the page:

  • Unable to check product pages: changes may take 24 to 72 hours to show on the Needs attention page. After a new feed upload or Merchant API update, errors should disappear within a couple of days. The page says increasing Google's crawl rate can speed this up.
  • Product crawl issues: after the fix, it may take up to 48 hours for the product to reappear in listings. For certain issues you can request a website check on the product details page under Needs attention.
  • Can't access product images: errors should disappear within a couple of days once robots.txt is fixed.
  • Robots.txt error (3D assets): Google's crawlers find the updated file automatically, and the page links to instructions for asking Google to refresh its cached copy sooner.

To fix products in Merchant Center, Google gives two routes on the Unable to check product pages page: one at a time, open each affected product, click Edit product and save after fixing robots.txt; or in bulk, download the affected products as a .csv file, cross-reference it with your product data and re-upload the corrected data. For image errors, the image page says no further action is needed once robots.txt is fixed.

Frequently asked questions

Does Unable to check product pages always mean robots.txt?

No. Google names robots.txt blocks and HTTP errors on the landing page as causes, and its product crawl issues page adds 404s, pages that require authentication, timeouts, too many redirects, firewalls and geo-restrictions.

Do I have to copy Google's exact robots.txt example?

No. Google's image help page says the example is not required and any setup is acceptable as long as it does not block Google's crawlers. Keep your existing private-path rules, but make sure product and image URLs are allowed for Googlebot and Googlebot-image.

Should I allow Storebot-Google as well?

These Merchant Center help pages only ask for Googlebot and Googlebot-image. Google's crawler documentation separately lists Storebot-Google as the token whose rules affect Google Shopping surfaces, so it is worth checking that it is not blocked either.

How long until disapproved products come back?

Google's pages give 24 to 72 hours for the Needs attention page to update and up to 48 hours for a fixed product to reappear in listings. Google says increasing its crawl rate, or requesting a website check where the product details page offers one, can speed this up.

Check a product page from the outside

Once robots.txt and the firewall are fixed, run the free check on one affected product URL to confirm the page returns a normal response and is not blocked for the main search and AI crawlers. If many products are affected across templates, hosts and regions, an audit reviews robots.txt, WAF behaviour and redirects across the store with evidence for each fix.

Filed under Platform guides. Topics: ecommerce, google, robots-txt, waf, troubleshooting