Canonical tags and duplicate URLs: how they confuse crawlers

A canonical tag tells crawlers which URL is the main version of a page when the same content is reachable at several addresses. It is a hint, not a command: Google treats rel="canonical" as a strong signal, but it also weighs redirects, sitemaps and HTTPS, and it may pick a different URL when those signals disagree. Most canonical problems are not missing tags but conflicting ones, and most conflicts can be found with a few curl commands and Search Console's URL Inspection tool.

Facts come from Google Search Central and RFC 6596, read on 8 November 2026.

What does a canonical tag actually signal?

A canonical tag is a link element in the page head that names the preferred URL for the content:

<head>
  <link rel="canonical" href="https://www.example.com/shoes/trail-runner">
</head>

RFC 6596, which defines the relation, says the target "MUST identify content that is either duplicative or a superset" of the page that declares it. In plain terms: the canonical page has to contain everything the duplicate has. A canonical from a product page to the homepage breaks that rule, and the RFC says explicitly that page 2 of an article "SHOULD NOT" name page 1 as its canonical, because page 1 does not contain page 2's content.

Google's canonicalization overview describes the process as clustering pages with the same or very similar primary content and choosing the one that is most complete and useful as the canonical. It calls your stated preference "a hint, not a rule." Google's guide to consolidating duplicate URLs orders the ways to state that preference by how strongly they influence the choice:

The same guide adds placement rules worth memorising. The link element is "only accepted if it appears in the <head> section of the HTML", absolute URLs are recommended over relative ones, and the canonical page should carry a self-referencing canonical. Google also suggests picking either the element or the header, because using both is more error prone. For a PDF, the header form looks like this:

HTTP/1.1 200 OK
Content-Type: application/pdf
Link: <https://www.example.com/downloads/white-paper.pdf>; rel="canonical"

Why canonicals matter for AI search

For Google, the link is direct. To appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show in Search with a snippet, according to Google's AI features documentation. Google uses the canonical page as the main source to evaluate content, and a search result usually points to it. If Google accepts a canonical that points at the wrong place, the wrong URL is the one it evaluates and usually shows. See Google AI Overviews: the technical requirements for the rest of that list.

For other AI search tools, the picture is less documented. At the time of writing, OpenAI's crawler documentation, Anthropic's crawler help article and Perplexity's crawler documentation say nothing about how their crawlers treat canonical tags.

Duplicates also waste crawling, which Google lists as one reason to declare a canonical. AI crawlers waste fetches too: in a December 2024 analysis of crawler traffic on nextjs.org, Vercel and MERJ found that 34.82% of ChatGPT's fetches hit 404 pages and a further 14.36% hit redirects, against 8.22% 404s for Googlebot. That data is about dead and redirected URLs, not canonicals, but the lesson carries over: make one URL per page the only one you link, list and serve.

Where duplicate URLs come from

Google's overview lists the usual sources: region variants, device variants, HTTP and HTTPS, sorted or filtered pages, and accidental copies such as a demo site left open to crawlers. In practice, these are the ones an audit finds most often:

Google's own example of a duplicate is a product URL reached through https://example.com/dresses/cocktail?gclid=ABCD. A self-referencing canonical in the page template handles this whole class of problem, as long as the template builds the URL from the clean path rather than echoing the requested URL: whatever parameters a link adds, the tag names the clean URL.

Trailing slashes deserve care because frameworks handle them differently. The Next.js trailingSlash documentation, for example, says Next.js redirects /about/ to /about by default, and the trailingSlash: true option reverses that. Whichever form your server redirects to is the form your canonical tags, sitemap and internal links must use.

The conflicts that make crawlers ignore a canonical

Google's consolidation guide warns: "Don't specify different URLs as canonical for the same page using different canonicalization techniques." These are the conflicts that cause it.

Canonical vs redirect

Page A declares B as canonical, but B redirects back to A, or redirects somewhere else. RFC 6596 advises avoiding a canonical target that is "the source IRI of a permanent redirect." Point the canonical at the final URL, the one that returns 200. Redirect chains and loops are covered in status codes, redirects and AI crawlers.

Canonical vs sitemap

The sitemap lists https://example.com/shoes/ while the page says its canonical is https://www.example.com/shoes. Google names exactly this mismatch, one URL in the sitemap and another in rel="canonical", as something to avoid. Generate your sitemap from the same source as your canonical tags so they cannot drift. More on that in XML sitemaps for AI discovery.

Canonical vs internal links

Your navigation links to /shoes?ref=nav or to the non-www host, while the canonical says otherwise. Google says that linking consistently to the canonical URL helps it understand your preference, and internal links are also how crawlers find pages in the first place, so link to the canonical form everywhere. See internal links and broken links.

Canonical vs hreflang

A site with English, German and French versions sets every language page's canonical to the English page. That tells Google the German and French pages are duplicates, while hreflang says they are alternates. Google's guidance is to specify a canonical in the same language, or the best substitute language if one does not exist. Each language version should declare itself as canonical and list the others with hreflang. Google also notes that language versions count as duplicates only when the primary content is in the same language, so translating only the header and footer does not make two pages distinct.

Canonical vs noindex

Putting noindex on a page and a canonical pointing elsewhere sends two different messages. Google says it does not recommend noindex "to prevent selection of a canonical page within a single site", because noindex blocks the page from Search entirely. Use one or the other: canonical for duplicates, noindex for pages that should never appear.

Canonical vs robots.txt

If robots.txt blocks a duplicate URL, crawlers that obey it never fetch the page, so they never see its canonical tag. Google says plainly: "Don't use the robots.txt file for canonicalization purposes," and adds that it may still index disallowed URLs without their content.

Canonicals that point to the wrong place

Some canonical tags are not conflicting, just wrong.

• A 404 or 410 target. RFC 6596 advises avoiding a target that "returns an error code, such as a 4xx response." This happens after a URL change when templates keep the old path.
• A staging or preview host. A SITE_URL variable left at its staging value makes every live page declare https://staging.example.com/... as canonical. The leftover staging settings guide covers this alongside the other launch-day leaks.
• One URL for every page. A theme that prints the homepage URL as the canonical on every page tells search engines the whole site is a duplicate of the homepage.
• Chained canonicals. A points to B, B points to C. RFC 6596 advises avoiding a target that itself declares a canonical to another URL. Point A straight at C.
• Two canonical tags. A plugin and a theme both add one, with different URLs. RFC 6596 says to specify only one canonical for a resource. Remove the duplicate source.
• Set or changed by JavaScript. Google asks you to put the canonical in the HTML source and make sure JavaScript does not change it. Crawlers that do not run JavaScript see only what the server sent.

Cross-domain canonicals and syndication

RFC 6596 allows a canonical on a different domain, and that is legitimate when you move content between sites you own or run the same catalogue on two domains. Two cautions apply.

First, for syndicated articles that a partner republishes, Google's canonicalization troubleshooting guide says the canonical element is not recommended, because the pages are often very different. It says "the most effective solution is for partners to block indexing of your content", for example with noindex on their copy.

Second, the same guide lists injected cross-domain canonicals as a sign of malicious hacking: an attack adds a canonical, in the HTML head or an HTTP header, pointing to a spammy URL, and Google may choose that URL instead of the compromised page. If an audit shows a canonical to a domain you do not recognise, treat it as a security issue, not an SEO setting.

How to audit canonical tags

You can check a single page with curl in a few minutes. For more on reading raw responses, see how to see your page as an AI crawler, with curl.

1. Read the canonical from the raw HTML and headers.

   bash

   URL="https://www.example.com/shoes/trail-runner"
   curl -sL "$URL" | grep -Eio '<link[^>]*rel=["'"'"']?canonical[^>]*>'
   curl -sIL "$URL" | grep -i '^link:'

   You want exactly one canonical, with an absolute URL. The grep reads the HTML as the server sent it, so a canonical added by JavaScript will not show up here.

2. Check that the canonical target returns 200 with no redirect. Set CANON to the URL the tag names.

   bash

   CANON="https://www.example.com/shoes/trail-runner"
   curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "$CANON"

   You want 200 and nothing after it. A 301, 302, 404, or any redirect URL printed after the status code means the canonical needs updating.

3. Check that the target names itself. Run step 1 on the canonical URL. If it declares a different canonical, you have a chain.

4. Test the duplicates. Request the page with a tracking parameter, a trailing slash, the other protocol and the other host. Each should either redirect to the canonical URL or declare it as canonical.

   bash

   for u in "$URL?utm_source=test" "$URL/" "http://www.example.com/shoes/trail-runner" "https://example.com/shoes/trail-runner"; do
     echo "== $u"
     curl -s -o /dev/null -w '%{http_code} -> %{redirect_url}\n' "$u"
   done

   For any variant that returns 200, run step 1 on it and confirm the canonical names the clean URL.

5. Compare with the sitemap, hreflang and internal links. The sitemap entry, each hreflang href that refers to this page, and your navigation links should all use the exact canonical string, including protocol, host, case and trailing slash.

6. Ask Google what it chose. The URL Inspection tool in Search Console shows the user-declared canonical and the Google-selected canonical. If they differ, look for the conflicting signals above. Google's troubleshooting guide notes that, even after you fix content issues, it may keep pages in a duplicate cluster for up to two weeks.

For a whole site, run steps 1 to 3 across every URL in your sitemap and flag any page where the canonical is missing, relative, off-host, non-200 or different from the sitemap entry.

Frequently asked questions

Does every page need a self-referencing canonical tag?

Google recommends one on every canonical page, and it is the simplest defence against parameter and tracking-link duplicates. It costs one line in the template and removes ambiguity for every crawler.

Is a canonical tag better than a 301 redirect?

They do different jobs. Use a 301 when the duplicate URL should stop existing, and a canonical tag when the duplicate must stay reachable, such as a filtered list or a URL with tracking parameters.

Can Google ignore my canonical tag?

Yes. Google calls it a hint, not a rule, and may choose another URL when redirects, sitemaps, HTTPS or content differences point elsewhere. URL Inspection in Search Console shows which URL it picked.

Do ChatGPT, Claude and Perplexity respect canonical tags?

None of their crawler documentation mentions canonical tags at the time of writing, so their handling is not documented. Make every signal agree on one URL so that no crawler has to choose.

Should translated pages canonical to the original language?

No. Google advises a canonical in the same language, so each language version should point to itself and link its alternates with hreflang.

Next step

The free check tests whether a single page's canonical is present and self-consistent. Duplicate URLs without a shared canonical only show up across many pages, which is what a full-site audit covers, for up to 500 pages with the evidence for each finding.