WooCommerce SEO for AI crawlers: product pages, schema, robots.txt
Make WooCommerce product pages readable for AI crawlers: built-in Product schema, robots.txt for cart and filter URLs, and plugin conflicts that block bots.
By Crawlito · 6 min read
In short
- WooCommerce renders product pages on the server in PHP, so the product name, price and description are in the HTML crawlers receive unless a theme or plugin moves them into JavaScript.
- WooCommerce core outputs Product JSON-LD with offers, SKU, availability and reviews, but it does not add a brand property, which you can add with the woocommerce_structured_data_product filter.
- WooCommerce adds a noindex robots meta tag to its cart, checkout and account pages by default.
- Add-to-cart links, sorting and layered-navigation filter parameters create many crawlable URLs; Google recommends blocking unneeded faceted URLs in robots.txt.
- Security and caching plugins are the usual reason a WooCommerce store blocks crawlers or serves them stale prices.
On this page
WooCommerce is in a good starting position for AI crawlers: product pages are rendered on the server, so the name, price, description and stock status are in the HTML a crawler receives, and WooCommerce core adds Product structured data as JSON-LD. What usually goes wrong is everything around the product page: crawlers wandering through add-to-cart and filter URLs, a security plugin that blocks bot user agents, or a cache that serves an old price.
This guide covers what WooCommerce does by default, what to add, which URLs to keep crawlers out of, and how to check each part on your own store. For general WordPress settings such as "Discourage search engines", start with our WordPress AI search readiness guide.
Is WooCommerce product content visible to crawlers?
Usually yes. WooCommerce templates are PHP, so a product page's title, price, short description, long description and attributes are printed into the HTML before it leaves the server. A crawler that does not run JavaScript gets the same core text a browser shows.
The exceptions come from what you add on top:
- Block or page-builder themes that load product tabs, descriptions or reviews after the page loads.
- Variation prices for variable products. The price range is in the HTML, but the exact price for a chosen variation is updated in the browser when a shopper picks options.
- Third-party widgets for reviews, size guides, specs or Q&A that fetch content with JavaScript.
- "Load more" or infinite scroll on category pages, which can hide products beyond the first batch. See pagination and infinite scroll.
To check a product page, fetch it without a browser and look for a sentence from the description and the price:
curl -s -A "GPTBot/1.1" https://shop.example.com/product/blue-mug/ | grep -c "Hand-glazed stoneware"
curl -s -A "GPTBot/1.1" https://shop.example.com/product/blue-mug/ | grep -o 'woocommerce-Price-amount[^<]*<[^>]*>[^<]*' | head -3If the description count is zero, that content is being added after load and crawlers that skip JavaScript will not see it.
What structured data WooCommerce outputs by default
WooCommerce core has a structured data class, WC_Structured_Data, that collects markup while the page is built and prints it as one JSON-LD block in the footer. On a single product page you typically get:
| Type | Where | Main properties |
|---|---|---|
Product | Single product pages | name, url, description, image, sku (falls back to the product ID), gtin when a valid one is set |
Offer or AggregateOffer | Inside Product | price or lowPrice/highPrice, priceCurrency, priceValidUntil for scheduled sales, availability (InStock, OutOfStock, BackOrder), seller |
AggregateRating and Review | Inside Product, when approved reviews exist | Rating value, review count, recent reviews |
BreadcrumbList | Pages that show WooCommerce breadcrumbs | Breadcrumb trail |
WebSite with SearchAction | Shop page when it is the front page | Site search |
You can read the exact logic in WooCommerce's source on GitHub, which is the most reliable reference because the markup changes between versions.
Two gaps are worth knowing:
- No
brand. Core does not map a brand property. Google listsbrandas a recommended property for product markup, so stores often add it. - GTIN only when filled in. Recent WooCommerce versions read a global unique ID field on the product (GTIN, UPC, EAN or ISBN) and output
gtinonly if the value is valid. Empty fields produce nogtin.
Adding brand with a filter
The woocommerce_structured_data_product filter receives the markup array and the product. Extend it rather than replacing it, so price and availability stay in sync with core:
add_filter( 'woocommerce_structured_data_product', function ( $markup, $product ) {
$brand = $product->get_attribute( 'pa_brand' );
if ( $brand ) {
$markup['brand'] = array(
'@type' => 'Brand',
'name' => $brand,
);
}
return $markup;
}, 10, 2 );Adjust pa_brand to the attribute or taxonomy your store uses for brands.
Avoiding duplicate Product markup
SEO plugins and schema plugins often output their own Product markup. Two Product entities for the same item, with different prices or ratings, are confusing for any consumer of the data. Pick one source: either let the plugin replace WooCommerce's markup (most document a setting for this) or turn the plugin's product schema off. Then validate with Google's Rich Results Test. Our Product schema guide explains which properties Google uses and why the marked-up price must match the visible one.
Which WooCommerce URLs to keep crawlers out of
A WooCommerce store generates many URLs that are not useful to crawl:
| URL pattern | Where it comes from | Default handling |
|---|---|---|
/cart/, /checkout/, /my-account/ | WooCommerce pages | WooCommerce adds a noindex robots meta tag |
?add-to-cart=123 | "Add to cart" buttons on archive pages | Crawlable unless you block it |
?orderby=price | Sorting dropdown | Crawlable |
?filter_color=blue&query_type_color=or | Layered navigation filters | Crawlable |
?min_price=10&max_price=50 | Price filter | Crawlable |
The add-to-cart links matter most. They appear on every category and shop page, and each request can add an item to a session cart, which costs server work and can create sessions for bots.
Google's guidance on faceted navigation says that if you do not need filtered URLs in search, you should prevent their crawling, and robots.txt is the most direct way. A starting point for a WooCommerce store:
User-agent: *
Disallow: /*?add-to-cart=
Disallow: /*&add-to-cart=
Disallow: /*?orderby=
Disallow: /*?filter_
Disallow: /*&filter_
Disallow: /*?min_price=
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Sitemap: https://shop.example.com/wp-sitemap.xmlNotes on this file:
- Check your permalinks. If your cart page is
/basket/or your store uses a different language, adjust the paths. - Keep filters you want found. If a filtered page such as "blue mugs" is a real landing page, give it a clean URL (a category or tag) rather than relying on a filter parameter.
- Repeat rules in named groups. If you add a
User-agent: GPTBotorUser-agent: PerplexityBotgroup, it replaces the*group for that crawler, so copy these lines into it. - Robots.txt and noindex interact. A crawler that is disallowed from
/cart/cannot see its noindex tag. For cart and checkout that is fine, because neither page has anything worth indexing. - Your sitemap URL may differ. WordPress core serves
wp-sitemap.xml; SEO plugins often replace it with their own.
WordPress generates robots.txt virtually, so to edit it you either use your SEO plugin's editor or upload a physical file, which overrides the virtual one. Our guide to faceted navigation crawl traps has more patterns.
Security plugins that block crawlers
WordPress security plugins and host-level firewalls are the most common reason a WooCommerce store is invisible to AI crawlers. Bot-blocking lists, "block fake Googlebots" options, country blocking and rate limits can all catch legitimate crawlers. Store owners tend to tighten these after a card-testing attack on checkout, and the rules end up applying site-wide.
Protect what needs protecting, which is checkout, account and login endpoints, and leave product and category pages reachable. Then test with crawler user agents:
for ua in "GPTBot/1.1" "OAI-SearchBot/1.0" "PerplexityBot/1.0" "ClaudeBot/1.0"; do
printf "%-18s " "${ua%%/*}"
curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://shop.example.com/product/blue-mug/
doneAnything other than 200 needs explaining. Our guide to WordPress security plugins that block crawlers walks through the common settings.
Caching without serving stale prices
Page caching makes WooCommerce fast for crawlers and shoppers, but it must leave the dynamic parts alone. WooCommerce's caching guidance says to exclude the Cart, My Account and Checkout pages from caching, and lists cookies you can exclude, including woocommerce_cart_hash, woocommerce_items_in_cart and wp_woocommerce_session_.
For crawlers, the bigger risk is the opposite problem: product pages cached for so long that a price or stock change does not reach the HTML. If a sale ends but the cached page still shows the sale price, the visible price and the JSON-LD price are both stale, and anything a crawler stores is out of date. Make sure your cache purges a product page when the product is saved, including scheduled sale start and end times. See CDN cache and stale content for how to check.
Frequently asked questions
Does WooCommerce add Product schema automatically?
Yes. WooCommerce core outputs Product JSON-LD on single product pages, including offers, SKU, availability and reviews where they exist. It does not include brand, which you can add with the woocommerce_structured_data_product filter.
Should I block add-to-cart URLs in robots.txt?
Usually yes. They are action URLs, not content, and blocking them with Disallow: /*?add-to-cart= stops crawlers from triggering cart operations while browsing category pages.
Does WooCommerce noindex the cart and checkout pages?
Yes. WooCommerce adds a noindex robots meta tag to its transactional pages such as cart, checkout and my account.
Are variable product prices visible to crawlers?
The price range is in the server HTML and in the AggregateOffer markup. The exact price for a selected variation is updated in the browser, so crawlers see the range, not each combination.
Check a product page
The quickest way to know what crawlers get from your store is to test a real product URL. Run the free check on one, or book an audit to review product pages, robots.txt and plugin settings across the store.
Filed under Platform guides. Topics: wordpress, ecommerce, structured-data, robots-txt, ai-crawlers