Skip to content

Cloudflare Pages pages.dev URLs: stop duplicate indexing

Cloudflare Pages documents noindex on preview deployments only, not your production pages.dev URL. How to redirect it, add a _headers rule, and test it.

By Crawlito · 8 min read

In short

  • Cloudflare documents that every Pages preview deployment carries an X-Robots-Tag: noindex header, but it describes no such header for the production project.pages.dev address.
  • Cloudflare documents an account-level Bulk Redirect with a 301 to the custom domain for the production pages.dev URL, and its _redirects file does not support domain-level redirects.
  • A _headers rule on https://:project.pages.dev/* can add noindex instead, but _headers is not applied to responses generated by Pages Functions.
  • Preview deployment URLs are public by default, and enabling the Cloudflare Access policy restricts viewing them to your Cloudflare account.
  • Google calls redirects and rel=canonical strong canonical signals and does not recommend noindex for choosing a canonical page, because it blocks the page from Search completely.
On this page

Cloudflare Pages already keeps preview deployments out of search: Cloudflare documents that every preview deployment includes an X-Robots-Tag: noindex header. Your production address at <project>.pages.dev is a different case. Cloudflare's docs describe no such header for it, it stays reachable after you add a custom domain, and it serves the same content, so it can be crawled and indexed beside your real domain. The fix Cloudflare documents is a Bulk Redirect from <project>.pages.dev to the custom domain, with a _headers noindex rule as a fallback.

This post covers what each Pages hostname sends, the two fixes and their limits, and how to check the result with curl. Facts come from Cloudflare's Pages documentation and Google Search Central, read on 15 November 2026. For the wider set of staging leftovers on other hosts, see how staging settings hide live sites.

Which Pages hostnames get noindex by default

A Pages project answers on several hostnames at once. Cloudflare's preview deployments documentation describes the preview ones: each preview gets a hash-based address such as <hash>.<project>.pages.dev, and each branch gets an alias such as fix-api.<project>.pages.dev that always points at the branch's latest commit. Branch names are lowercased, and characters other than letters and numbers become hyphens.

HostnameWhat it servesDefault noindex header
<hash>.<project>.pages.devOne preview deploymentYes, X-Robots-Tag: noindex on every preview
<branch>.<project>.pages.devLatest preview of a branchDocumented for preview deployments; check it with curl
<project>.pages.devProduction deploymentNot documented
www.example.com (custom domain)The site you want indexedNot documented, and you should not want one

Cloudflare explains the header as a way to keep search engines from finding duplicate content. You can confirm it the way Cloudflare suggests, with curl -I against a preview URL.

What the docs do not say matters as much. Neither the preview page nor the custom domains page mentions a noindex header on the production pages.dev hostname. The custom domains page includes a section on how to "disable access" to the *.pages.dev subdomain, which tells you it stays live after a custom domain is added. Treat the production pages.dev URL as an indexable copy of your site until you have tested it.

Why an indexable pages.dev copy is a problem

When two hosts serve the same pages, search engines have to pick one. Usually they pick your custom domain, but nothing guarantees it, and the pages.dev address can turn up in results, in links people share, or as the canonical Google chooses. If that happens, Search Console reports the custom domain URL as a duplicate, the situation covered in duplicate, Google chose different canonical than user.

AI search tools inherit the same confusion. Google's AI features draw on its index, and assistants that cite pages link to whichever URL their own systems settled on. A citation pointing at project.pages.dev still works, but it sends people and any link value to a hostname you do not brand or control at DNS level.

It is the same kind of problem as www vs non-www, with one extra host. The difference is that you cannot add DNS records or zone-level rules to pages.dev, because Cloudflare owns that domain. The fixes have to come from account-level features or from files in your project.

Fix 1: redirect pages.dev to your custom domain

Google's canonicalization guide calls a redirect "a strong signal" that its target should become canonical. Cloudflare documents exactly this setup in redirecting pages.dev to a custom domain, using Bulk Redirects, which work at account level.

You cannot do it in the project's _redirects file. The Pages redirects documentation lists domain-level redirects as not supported, and notes that _redirects rules are not applied to requests served by Pages Functions.

The steps, from Cloudflare's guide:

  1. In Workers & Pages, open the project and confirm the custom domain is listed under Custom domains.
  2. Go to Bulk Redirects and create a redirect list.
  3. Add a list item with the values below.
  4. Create a Bulk Redirect rule that uses the list.
  5. Visit <project>.pages.dev and confirm you land on the custom domain.
SettingValue in Cloudflare's guide
Source URL<project>.pages.dev
Target URLhttps://example.com (your custom domain)
Status301
ParametersPreserve query string, Subpath matching, Preserve path suffix, Include subdomains

Watch the Include subdomains option

Cloudflare's Bulk Redirects parameter reference says that with Include subdomains on, the source hostname "will also apply to any subdomains". Preview URLs such as <hash>.<project>.pages.dev are subdomains of the source, so by that definition they would redirect to production too. If your team needs working preview links, test one after the rule propagates. If it redirects and you want it not to, turn Include subdomains off for that list item and protect previews with Cloudflare Access instead (covered below).

Subpath matching with Preserve path suffix is what keeps deep URLs intact: <project>.pages.dev/pricing/ goes to /pricing/ on the custom domain, not to the homepage.

Fix 2: add noindex with a _headers rule

If you cannot redirect, for example because something still depends on the pages.dev address, you can send a noindex header from it instead. The Pages headers documentation gives this example for a _headers file, a plain text file with no extension placed in your static asset folder (for example public/ in many frameworks, or the build output folder without one):

https://:project.pages.dev/*
  X-Robots-Tag: noindex

https://:version.:project.pages.dev/*
  X-Robots-Tag: noindex

The :project and :version placeholders match any project name and any preview or branch prefix. Absolute URLs in _headers must begin with https, cannot name a port, and match regardless of the request's protocol and port. Because the rules name pages.dev hosts only, your custom domain is not affected. At the time of writing a _headers file can hold up to 100 rules, with 2,000 characters per line.

This is a fallback, not the first choice. Google's canonicalization guide says it does not recommend noindex to prevent selection of a canonical page, because it completely blocks the page from Search, and it calls rel="canonical" link annotations the preferred solution. That advice is written for duplicates within one site, but the reasoning carries over. A redirect does both jobs: it removes the copy and points at the right URL. If you use the header, also make sure every page carries an absolute canonical to the custom domain, as described in the canonical tags guide.

The Pages Functions limit

Cloudflare states that headers in _headers "are not applied to responses generated by Pages Functions". If your site renders pages through Functions, or a framework adapter that runs on Functions, the header will be missing on exactly the HTML pages you care about. Once a project has Functions, the routing documentation says all requests invoke them by default unless _routes.json excludes those paths.

In that case set the header in code. A root-level functions/_middleware.js runs in front of every route, static files included, according to the middleware documentation. A minimal version:

js
// functions/_middleware.js
export async function onRequest(context) {
  const response = await context.next();
  const host = new URL(context.request.url).hostname;
  if (!host.endsWith(".pages.dev")) return response;
  const copy = new Response(response.body, response);
  copy.headers.set("X-Robots-Tag", "noindex");
  return copy;
}

The host check matters. Without it, the header would reach your custom domain and remove the live site from search, the leak described in noindex detected in X-Robots-Tag header.

Lock preview deployments with Cloudflare Access

The noindex header keeps previews out of search results, but Cloudflare notes that preview URLs are public by default. Anyone with a link can open them, and a noindex header does not stop a crawler from fetching the page. If previews contain unreleased work, restrict them:

  1. In Workers & Pages, select the project.
  2. Go to Settings > General and select Enable access policy.

Cloudflare says this restricts viewing deployments to your Cloudflare account (you can add team members to the account) and protects only preview deployments, not the production *.pages.dev address or custom domains. That is why the custom domains page pairs it with the Bulk Redirect for production.

Why robots.txt is the wrong tool here

It is tempting to block pages.dev with Disallow: /. Two things stop that working.

First, a Pages deployment serves the same files on every hostname it answers on, so a static /robots.txt in your build is the same file on <project>.pages.dev and on your custom domain. A Disallow: / meant for pages.dev would block the live site too. robots.txt applies per host, protocol and port, as explained in does robots.txt apply to subdomains, but with one static file you cannot give the two hosts different answers.

Second, Google's block indexing guide says a page must not be blocked by robots.txt for noindex to work, because the crawler never sees the rule, and a blocked URL can still appear in results through links. Blocking pages.dev would hide both your noindex header and your redirect from Google.

If you use the Bulk Redirect, the robots.txt question disappears: <project>.pages.dev/robots.txt redirects to the custom domain's file, and RFC 9309 says crawlers apply a redirected robots.txt in the context of the original host.

How to check each hostname

Run these from a terminal and compare the answers:

bash
# Production pages.dev: expect a 301 to the custom domain, or an x-robots-tag line
curl -sI https://project.pages.dev/pricing/ | grep -iE '^(HTTP|location|x-robots-tag)'

# A preview: expect x-robots-tag: noindex (or a login if Access is on)
curl -sI https://abc123.project.pages.dev/ | grep -iE '^(HTTP|location|x-robots-tag)'

# Custom domain: expect 200 and no x-robots-tag noindex
curl -sI https://www.example.com/pricing/ | grep -iE '^(HTTP|x-robots-tag)'
curl -s https://www.example.com/pricing/ | grep -io '<link[^>]*canonical[^>]*>'

Test a deep URL, not only the homepage, and test a page rendered by Functions if your project has any. Then, in Google Search Console for the custom domain, use URL Inspection on a key page and check that the Google-selected canonical is the custom domain URL. After the fix, Google only drops pages.dev URLs once it recrawls them, which its block indexing guide says can take months for less important pages.

Frequently asked questions

Does Cloudflare Pages add noindex to pages.dev?

Cloudflare documents an X-Robots-Tag: noindex header on every preview deployment. It documents no such header for the production <project>.pages.dev address, so test it with curl -I and redirect or noindex it yourself.

Can I redirect pages.dev to my domain with the _redirects file?

No. Cloudflare's redirects documentation lists domain-level redirects as unsupported in _redirects. Use an account-level Bulk Redirect, as Cloudflare's own guide does.

Why is my _headers noindex rule missing on some pages?

Headers from _headers are not applied to responses generated by Pages Functions. Set the header in functions/_middleware.js or in the Function itself, and limit it to pages.dev hostnames.

Will the Bulk Redirect also redirect my preview deployments?

With the Include subdomains parameter on, Cloudflare says the rule also applies to subdomains of the source host, which covers preview hostnames. Test a preview URL after enabling the rule and turn the option off if you need previews to load.

Should I block pages.dev in robots.txt?

No. The same robots.txt file is served on every hostname of the deployment, so a block would also hit your custom domain, and a blocked crawler cannot see your redirect or noindex header.

Check what crawlers get on your live domain

After the redirect or header is in place, confirm the custom domain itself is clean. The free check looks at a single public page with no account needed. For a whole site, an audit covers up to 500 pages and delivers a PDF with evidence and prioritized fixes.

Filed under Platform guides. Topics: cloudflare, noindex, canonical, deployment