Cloudflare Redirects for AI Training: canonicals become 301s
How Cloudflare Redirects for AI Training turns rel=canonical tags into 301s for verified AI training crawlers, who is excluded, plans, limits and how to check it.
By Crawlito · 9 min read
In short
Redirects for AI Training answers verified bots in Cloudflare's AI Crawler category with a 301 to the page's rel=canonical URL when that URL is same-origin and differs from the requested URL.
Browsers, search engines, AI Search bots and AI Assistant bots get the original page, so the feature changes nothing for Googlebot, OAI-SearchBot or Perplexity-User.
Cloudflare only evaluates text/html origin responses and only finds a canonical tag within the first 256 KB of the uncompressed body; cross-origin and self-referencing canonicals pass through.
The feature is on Pro, Business and Enterprise plans at no extra cost, and runs after the origin responds, so Single Redirects and Bulk Redirects that match a request always win.
A wrong canonical now has a direct cost: with the feature on, training crawlers are sent to whatever URL the tag names and never receive the page they asked for.
Cloudflare Redirects for AI Training takes the rel="canonical" tag already in your HTML and enforces it as a 301 Moved Permanently for one audience: verified bots in Cloudflare's AI Crawler category, such as GPTBot and ClaudeBot. If a page's canonical points to a different URL on the same origin, those crawlers are sent there instead of receiving the duplicate or deprecated page. Everyone else, including browsers, search engines and AI search crawlers, gets the original page unchanged.
That makes it a useful tool for sites with old docs versions, parameter duplicates or retired pages that must stay online. It also means your canonical tags stop being hints for this group of crawlers and start acting as redirects, so a wrong canonical now has a visible cost. Details below come from Cloudflare's Redirects for AI Training reference, the changelog entry of 17 April 2026 and Cloudflare's launch post, read on 15 November 2026. For canonical tags in general, see canonical tags and duplicate URLs; this post covers only what Cloudflare does with them.
What Redirects for AI Training does
Cloudflare's reference describes four steps for each response to a qualifying crawler:
Stream-parse the <head> of the origin's HTML response, looking for a <link rel="canonical"> tag.
Resolve a relative canonical URL against the request URL.
Check that the canonical URL is same-origin and is not the URL that was requested.
If both checks pass, return a 301 to the canonical URL.
If there is no canonical tag, the canonical points to another origin, or the page names itself, the origin response passes through unchanged.
A page such as this one, served at https://docs.example.com/v1/storage/:
html
<head>
<title>Storage API (v1, deprecated)</title>
<link rel="canonical" href="/v3/storage/">
</head>
would reach a verified training crawler as a redirect instead of the page:
http
HTTP/1.1 301 Moved Permanently
Location: https://docs.example.com/v3/storage/
The reference documents the 301 status and the canonical as its target, and the launch post shows the exchange with a Location header as above. Neither lists any other response headers, so do not build tooling that expects a particular extra header on the redirect.
Cloudflare's launch post gives the reason it built this. Its own deprecated Wrangler v1 docs carried a deprecation banner, a noindex tag and canonicals pointing to current docs, yet AI training crawlers kept consuming the old content at about the same rate as the current pages. The post reports 4.8 million visits to developers.cloudflare.com by bots in the AI Crawler category over 30 days. Advisory signals were not enough, so the canonical became an enforced pointer for that one class of crawler.
Which crawlers are redirected and which are not
Only verified bots with the AI Crawler category are redirected. On Cloudflare's verified bots page, that category is defined as bots that crawl websites for content used to train AI models. The launch post names GPTBot, ClaudeBot and Bytespider as examples.
At the time of writing, AI Crawler sits in what Cloudflare now calls its legacy categories. Since 1 July 2026 Cloudflare also classifies bots by behavior (Search, Agent, Training and others), and a bot can have more than one. The legacy categories still work in WAF custom rules, and the Redirects for AI Training reference still links to the AI Crawler category, so that is the rule this post describes.
Visitor
Category or type
What it gets
GPTBot, ClaudeBot, Bytespider
Verified, AI Crawler
301 to the canonical URL
OAI-SearchBot
Verified, AI Search (legacy)
Original page
Perplexity-User, DuckAssistBot
Verified, AI Assistant
Original page
Googlebot, Bingbot
Verified, Search Engine Crawler
Original page
Browsers
Human traffic
Original page
A script sending a GPTBot user agent
Not verified
Original page
The reference says only verified bots in the AI Crawler category are redirected, which leaves out any request that merely claims a training crawler's user agent. Two consequences follow:
AI search is untouched. Bots that Cloudflare classifies as AI Search or AI Assistant, such as OAI-SearchBot and Perplexity-User, keep getting the same pages as before. Which category a given crawler is in is Cloudflare's call, so check a bot you care about in Cloudflare Radar's bots directory rather than assuming.
It does not fix training data already collected. Cloudflare's launch post says plainly that the feature is not retroactive. It only affects future fetches.
Cloudflare's verified bots page also says it can reassign a bot's category if the bot's public documentation and observed behavior differ from the category in its submission. The set of crawlers that get the redirect therefore follows Cloudflare's classification, not a list you maintain.
Limits: HTML only, 256 KB, same origin
The reference lists six limitations. Two of them (verified AI Crawler bots only, and the loop safeguard) are covered elsewhere in this post; the other four decide whether a given page is eligible at all.
Condition
What Cloudflare documents
What it means for you
Content type
Only content-type: text/html responses are evaluated; other types pass through
A PDF that declares its canonical in a Link HTTP header is not redirected
Position
The tag must appear within the first 256 KB of the uncompressed body
A canonical after large inline scripts or styles is missed
Origin
Only same-origin canonicals trigger a redirect; cross-origin ones are ignored
A canonical to another domain passes through; so, by the usual web definition of origin, does one from example.com to www.example.com
Self-reference
Self-canonical pages are not redirected
Your canonical pages are always served normally
The launch post explains the cross-origin rule: those canonicals often serve domain consolidation rather than content freshness. The reference does not define same-origin in more detail, so treat a different scheme, host or port as a different origin unless you have tested otherwise.
Two things the reference does not address: which origin status codes are evaluated, and how query strings in the request or the canonical are handled. If your duplicates are mostly parameter URLs, check the logs described below before you rely on the redirect.
How it interacts with other redirects and Cloudflare features
Order matters, because Redirects for AI Training needs the origin's HTML before it can decide anything.
Single Redirects and Bulk Redirects run first. They act before the origin is contacted. If one matches, the crawler is redirected by that rule and Redirects for AI Training never sees the request.
Redirects for AI Training runs after the origin responds. It reads the origin HTML and only then decides whether to answer with a 301.
Markdown for Agents does not change the decision. If you also convert HTML to Markdown at the edge, the redirect decision is still based on the original HTML, and Markdown conversion applies only to responses that are not redirected.
For a URL that should be gone for everyone, a normal redirect is still the right tool: it applies to search engines too, which this feature deliberately leaves alone. 301, 302, 307 and 308 redirects covers which code to use. Redirects for AI Training fits the other case: a page that must stay reachable for people and search engines, such as an old docs version, while training crawlers are steered to the current one. The Docusaurus old docs version guide shows how those duplicates arise on one common docs platform.
The launch post adds why Cloudflare did not simply recommend Single Redirect rules matched on user agents: each new deprecated path needs a rule change, user agents have to be tracked by hand, and the rules use plan limits. Canonical tags already change with your content.
The loop safeguard
The reference describes best-effort loop detection based on the Referer header. If a crawler was just redirected from the canonical URL back to the current page, Cloudflare serves the origin HTML instead of redirecting again. This targets two-page misconfigurations, where page A names B as canonical and B names A. Because it relies on Referer, it is a safety net, not a substitute for fixing the tags.
How to turn it on
Cloudflare lists the feature on Pro, Business and Enterprise plans at no additional cost. There are three ways to enable it:
Whole zone, dashboard. Open AI Crawl Control for the zone and turn on Redirects for AI Training under Quick Actions.
Whole zone, API. Send a PATCH to the zone setting with a token that has Zone Settings edit permission:
Selected hosts or paths. Under Rules > Overview, create a Configuration Rule that matches the section, for example starts_with(http.request.uri.path, "/docs/") or http.host eq "docs.example.com", and set Redirects for AI Training to On.
Cloudflare for SaaS zones can enable it for all custom hostnames from Quick Actions. Enabling it for selected custom hostnames uses custom metadata and a matching Configuration Rule, which the reference says needs an advanced subscription.
Scoping with a Configuration Rule is the cautious start: enable it on the docs section where duplicates are deliberate, review the results, then widen it.
Why wrong canonicals now have a visible cost
Without this feature, a bad canonical is only a signal: Google's duplicate URL documentation calls rel="canonical" a strong signal among several, and other crawlers may not use it at all. With it on, verified training crawlers follow the tag every time. Audit your canonicals before enabling it, because these common mistakes turn into real redirects:
Every page canonical to the homepage. A theme bug like this would send training crawlers to the homepage from every page that carries the tag.
Paginated pages canonical to page 1. Pages 2 and onward would never be served to training crawlers.
Translations canonical to the main language. On a subdirectory setup, /de/ and /fr/ pages would redirect training crawlers to the English page.
A canonical that points to a 404 or a redirect. The crawler is sent somewhere that does not serve the content.
A canonical added late in a heavy <head>. Past the first 256 KB, it is silently ignored, so the feature does nothing for that page.
A note on serving bots different responses: here search engines get exactly what people get, and training crawlers get a redirect to content you publish openly. That is different from the content switching discussed in is serving AI crawlers different content cloaking?.
How to check it is working
You cannot test it with curl and a borrowed user agent, because your request is not a verified bot and will get the original page. Check the inputs and the logs instead.
Confirm the canonical is inside the first 256 KB. This reads the uncompressed HTML, cuts it at 256 KB and looks for the tag:
No output means no canonical in range, so no redirect.
Confirm the content type is HTML. Run curl -sI "$URL" | grep -i '^content-type' and look for text/html.
Confirm the target serves 200. Request the canonical URL and check it does not redirect or return an error.
Read the logs. When a redirect is issued, Cloudflare records the canonical target in the redirects_for_ai_training_target field of HTTP request logs, which you can query with the GraphQL Analytics API or Logpush. Filter for training crawlers and compare the targets with what you expected.
The Cloudflare AI crawler settings guide covers the rest of AI Crawl Control. A training crawler you block there gets a block response rather than your page, so this feature only matters for training crawlers you allow.
Frequently asked questions
Does Redirects for AI Training affect Google Search?
No. Cloudflare's reference says search engines receive the original page unchanged, and only verified bots in the AI Crawler category, such as GPTBot, are redirected.
Will it change what ChatGPT search or Perplexity sees?
No. AI Search bots such as OAI-SearchBot and AI Assistant bots such as Perplexity-User are excluded and get the original page.
Do I need to add new tags to my pages?
No. The feature uses your existing rel="canonical" tags, so the work is making sure those tags are correct, same-origin and within the first 256 KB of the HTML.
Is it available on the Free plan?
Not at the time of writing. Cloudflare lists it on Pro, Business and Enterprise plans at no additional cost.
Does it remove old pages from existing AI training data?
No. Cloudflare says the feature is not retroactive. It only changes what verified training crawlers receive on future requests.
Check your canonicals before you switch it on
Turning canonicals into redirects is only as safe as the canonicals themselves. Run the free check on a few important pages to see what AI crawlers can fetch, and if you have many versions or parameter duplicates, a full-site audit reviews canonical and crawler access issues across up to 500 pages with the evidence for each finding.
Filed under CDN and bot protection. Topics: · cloudflare, canonical, ai-training, ai-crawlers, redirects
Which Cloudflare settings block AI bots: AI crawler policies, AI Crawl Control, Bot Fight Mode, managed robots.txt and WAF rules, and how to allow the ones you want.
301 vs 302 vs 307 vs 308 explained: method rules from RFC 9110, where 303 fits, why Next.js sends 307 and 308, and the Chrome 307 that no crawler ever sees.
What Alternate page with proper canonical tag means in Search Console, when the count is fine, and the five template mistakes it can hide, with checks for each.