Unable to check product pages in Merchant Center: robots.txt fix
Why Merchant Center says it is unable to check product pages, how to find the robots.txt, firewall or HTTP error behind it, and how long re-crawling takes.
By Crawlito · 8 min read
In short
- Google says Unable to check product pages means it cannot crawl the landing page in the link attribute, because robots.txt blocks its crawler or the page returns an HTTP error.
- Affected products stay disapproved until Google can crawl the landing page, and Google's fix is to let both Googlebot (landing pages) and Googlebot-image (images) crawl the site.
- Google's product crawl issues page also lists 404s, 4xx and 5xx responses, pages that require a login, timeouts, too many redirects, firewalls, geo-restrictions and an unreachable robots.txt as causes.
- Google suggests allowing crawling from the US, since it often crawls from there, and asking your host to let the Googlebot user agent through the firewall.
- Google's pages give 24 to 72 hours for Needs attention to update and up to 48 hours for a fixed product to reappear in listings.
On this page
- The four Merchant Center crawl issues and what each means
- Fix robots.txt for Googlebot and Googlebot-image
- Look past robots.txt: HTTP errors, firewalls and region blocks
- How to find the block in five checks
- How long the re-crawl takes after a fix
- Frequently asked questions
- Check a product page from the outside
Merchant Center shows Unable to check product pages when Google cannot crawl the landing page you submitted in the link attribute. Google's help page for the issue names two causes: your robots.txt blocks Google's crawler, or the landing page returns an HTTP error when Google fetches it. The affected products stay disapproved and stop appearing in Shopping ads and free product listings until Google can crawl the page, so the fix is to find the block, remove it, and wait for the re-crawl.
You may also see the problem described as "page not crawlable due to robots.txt"; Google's current help page calls the issue Unable to check product pages. This post covers the crawl issues themselves. For a price disagreement between the feed and the page, see Merchant Center price mismatch after a sale. For Google's shopping crawler, which these help pages do not mention, see Storebot-Google.
The four Merchant Center crawl issues and what each means
Google spreads crawl problems across several help pages. They look alike in the Needs attention tab but point at different files:
| Issue in Merchant Center | What Google could not fetch | Crawler Google asks you to allow |
|---|---|---|
| Unable to check product pages | The landing page in link | Googlebot and Googlebot-image |
| Product crawl issues | Mobile and desktop landing pages and images, for many reasons beyond robots.txt | Googlebot and Googlebot-image |
| Can't access product images | Images in image_link | Googlebot and Googlebot-image |
| Robots.txt error | Information about 3D assets on your site | Googlebot |
The consequences differ. For landing pages, Google says a crawl error on either the mobile or the desktop page disapproves the item on both devices until it can access the page. For images, the help page says products are still shown for now but will soon be disapproved if robots.txt is not updated, because Merchant Center products require an image. The Robots.txt error page concerns 3D assets only and asks for a Googlebot group that allows the site.
Fix robots.txt for Googlebot and Googlebot-image
Google's example for the landing page and image issues is two groups with an empty Disallow, which allows everything:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:The image help page adds that this exact file is not required: any setup is fine as long as it does not block Google's crawlers. That matters, because pasting the example onto an existing file can change more than you intend.
Google's crawlers read only one group: the most specific one that matches them. Once you add a User-agent: Googlebot group, Googlebot stops reading your User-agent: * rules entirely. If your * group blocked /cart, /account or sort and filter URLs, those rules no longer apply to Googlebot. Repeat any you still want inside the new group, as explained in how robots.txt groups work.
Googlebot-Image has its own subtlety. If no group names it, it follows your Googlebot group, and only falls back to * when there is no Googlebot group. The details are in Googlebot-Image, Video and News tokens. A safe store file therefore keeps the private paths you need and makes sure product and image paths are open for both:
User-agent: *
Disallow: /account/
Disallow: /checkout/
User-agent: Googlebot
User-agent: Googlebot-image
Disallow: /account/
Disallow: /checkout/Before publishing, check three patterns that commonly catch product URLs by accident:
- Broad query blocks such as
Disallow: /*?, which also block/products/shirt?variant=123when your feed links to variant URLs. - Image or CDN paths such as
/media/or/cdn/blocked to save crawl budget, which trigger Can't access product images. - A separate image host. robots.txt is per host, so if
image_linkpoints atimages.example.com, that host's own/robots.txtmust allow Googlebot-image too.
Google's pages tell you to test with the URL Inspection tool in Search Console. Inspect one affected landing page URL exactly as it appears in the feed and read whether crawling is allowed.
Look past robots.txt: HTTP errors, firewalls and region blocks
Many stores have a clean robots.txt and still see the issue. Google's product crawl issues page lists the other causes it reports:
| Cause Google lists | What to check |
|---|---|
| Page not found (404) | The link value is current and the product still exists |
| HTTP 4xx and 5xx responses | The page returns a normal 200 to Google, not a 403, 429 or 503 |
| Page requires authentication | No login, password page or age gate in front of the product |
| Firewalls, geo-restrictions or private IP | WAF, bot protection and country rules let Googlebot through |
| Timeout reading page | The server answers quickly under crawler load |
| Server redirects too often, redirect errors | The URL resolves in a short chain to the product |
| Hostname not resolvable, network error | DNS and TLS work from outside your network |
| Malformed HTTP response | A proxy, CDN or app server is not sending broken headers or bodies |
| Robots.txt unreachable | /robots.txt itself returns 200 (or 404 if you have none), not a 5xx, a 429 or a timeout |
| Invalid URL | No invalid characters, and the link is a full valid URL |
| Regional landing pages not accessible | URLs with a region ID load for Google, if you use regional availability and pricing |
On firewalls, Google's advice is specific: allow crawling from the US, because Google often crawls from there, and ask your webmaster or hosting provider to update firewall and server settings to allow the Googlebot user agent. A store that blocks every country but its own, or challenges unknown traffic with a JavaScript page, can pass every manual test from the office and still fail for Googlebot. Geo rules are covered in geo blocking crawlers, and the 403 case in Googlebot blocked with 403 Forbidden.
Allowing "the Googlebot user agent" should not mean trusting any request that claims it. Configure the rule to allow requests that pass Google's crawler verification, a reverse and forward DNS lookup or a match against Google's published IP range files, so a spoofed user agent does not get a free pass.
If you use the regional availability and pricing feature, the product crawl issues page says all landing pages with a region ID must be crawlable. URLs such as https://mystorewebsite.com/p/myproductlandingpage?region_id=123456 need to load for Google as well as the plain product URL, so check that no robots.txt rule or redirect strips or blocks the region_id parameter.
How to find the block in five checks
Take one affected product and work through it in order. Each check rules out a layer.
- Copy the exact URL. In Products, open the Needs attention tab, filter for the issue, open the product and copy the
linkvalue. For many products, download the .csv list from the same tab. - Read robots.txt for that host. Open
https://yourstore.com/robots.txtand find the group Googlebot follows. Check everyDisallowagainst the URL path, including its query string. - Request the URL with a Googlebot user agent. This does not prove what Google sees, but it shows whether your own stack treats the string differently.
curl -sS -o /dev/null -w '%{http_code} %{url_effective} %{num_redirects}\n' -L \
-A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
'https://yourstore.com/products/blue-shirt?variant=123'A 403, 429 or 503, a long redirect chain, or a 200 that lands on a challenge or login page points at the server or WAF, not robots.txt.
- Check firewall and CDN events. Search your WAF or CDN logs for requests with
Googlebotin the user agent to that path. A block at the edge never reaches your origin logs. If you have region rules, check whether US traffic is challenged or refused. - Run URL Inspection. Use the live test for the same URL in Search Console; the URL has to be in a property you own. It reports Crawl allowed? and whether the Page fetch succeeded, and the tested page view shows the HTTP headers Google received.
Repeat the same steps for one image_link URL if the issue is Can't access product images.
How long the re-crawl takes after a fix
Google gives several timings, depending on the page:
- Unable to check product pages: changes may take 24 to 72 hours to show on the Needs attention page. After a new feed upload or Merchant API update, errors should disappear within a couple of days. The page says increasing Google's crawl rate can speed this up.
- Product crawl issues: after the fix, it may take up to 48 hours for the product to reappear in listings. For certain issues you can request a website check on the product details page under Needs attention.
- Can't access product images: errors should disappear within a couple of days once robots.txt is fixed.
- Robots.txt error (3D assets): Google's crawlers find the updated file automatically, and the page links to instructions for asking Google to refresh its cached copy sooner.
To fix products in Merchant Center, Google gives two routes on the Unable to check product pages page: one at a time, open each affected product, click Edit product and save after fixing robots.txt; or in bulk, download the affected products as a .csv file, cross-reference it with your product data and re-upload the corrected data. For image errors, the image page says no further action is needed once robots.txt is fixed.
Frequently asked questions
Does Unable to check product pages always mean robots.txt?
No. Google names robots.txt blocks and HTTP errors on the landing page as causes, and its product crawl issues page adds 404s, pages that require authentication, timeouts, too many redirects, firewalls and geo-restrictions.
Do I have to copy Google's exact robots.txt example?
No. Google's image help page says the example is not required and any setup is acceptable as long as it does not block Google's crawlers. Keep your existing private-path rules, but make sure product and image URLs are allowed for Googlebot and Googlebot-image.
Should I allow Storebot-Google as well?
These Merchant Center help pages only ask for Googlebot and Googlebot-image. Google's crawler documentation separately lists Storebot-Google as the token whose rules affect Google Shopping surfaces, so it is worth checking that it is not blocked either.
How long until disapproved products come back?
Google's pages give 24 to 72 hours for the Needs attention page to update and up to 48 hours for a fixed product to reappear in listings. Google says increasing its crawl rate, or requesting a website check where the product details page offers one, can speed this up.
Check a product page from the outside
Once robots.txt and the firewall are fixed, run the free check on one affected product URL to confirm the page returns a normal response and is not blocked for the main search and AI crawlers. If many products are affected across templates, hosts and regions, an audit reviews robots.txt, WAF behaviour and redirects across the store with evidence for each fix.
Filed under Platform guides. Topics: ecommerce, google, robots-txt, waf, troubleshooting