Site noindex by mistake: how staging settings hide live sites

If your site launched with a noindex by mistake, the cause is usually a setting that was correct on staging and travelled to production: a CMS checkbox, a header added by your host or CDN, a password, a Disallow: / in robots.txt, or canonical tags that still point at the staging domain. Each one tells search engines and AI crawlers to stay away, and none of them shows up when you look at the site in a browser. You can check all five from a terminal in about two minutes.

This guide walks through each leftover, how to confirm it, how to fix it, and a checklist to run on launch day. Facts about platforms and crawlers come from their own documentation, read on 8 November 2026.

Why a leftover noindex hides a site from AI search too

A noindex rule asks search engines to keep a page out of their index. Google supports it in two forms with the same effect, a robots meta tag and an X-Robots-Tag HTTP header, as its block indexing guide explains:

<meta name="robots" content="noindex">

HTTP/2 200
x-robots-tag: noindex

The AI connection is direct for Google. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Search with a snippet, according to Google's AI features documentation. A noindexed page fails that test.

Outside Google, support for noindex is documented unevenly, but the other leftovers on this list block them directly. A Disallow: / stops OAI-SearchBot, Claude-SearchBot and PerplexityBot, and a password page gives every crawler a login prompt instead of your content. For what each directive does, see noindex, nosnippet and X-Robots-Tag for AI search.

The five staging leftovers at a glance

1. WordPress "Discourage search engines from indexing this site"

WordPress has a checkbox under Settings > Reading > Search engine visibility that developers tick on staging. Since WordPress 5.3, ticking it outputs a noindex, nofollow robots meta tag in the head of every page. Up to 5.2 it served Disallow: / in the virtual robots.txt instead, as the Reading settings documentation notes. WordPress made the change because a robots.txt block could still let a linked URL appear in results.

The box is stored in the database, so it moves with the site whenever staging is pushed to production or a database is migrated. It also has a quieter effect: since WordPress 5.5, the core Sitemap: line is added to the virtual robots.txt only when the site is public, so a missing sitemap line is another clue. Keep in mind that the virtual robots.txt applies only when no physical robots.txt file exists.

Check it:

curl -s https://www.example.com/ | grep -io '<meta[^>]*robots[^>]*>'

A result containing noindex on your homepage is the warning sign. SEO plugins can also set noindex per page or per post type, so check a few page types, not only the homepage.

Fix it: untick the box, save, clear any page cache and CDN cache, and run the check again. The WordPress AI search guide covers the other settings worth reviewing at the same time.

2. X-Robots-Tag: noindex set by the host or CDN

Several hosts add a noindex header to preview URLs on purpose, which is good practice until the live site is served from one of those URLs.

• Vercel adds X-Robots-Tag: noindex to preview deployments and to production deployments requested through their unique deployment URL rather than a production domain, including outdated production deployments after a new one is promoted. Assigning a production domain to a preview removes the header, and Vercel does not add it to a preview served from a custom domain assigned to a Git branch, such as staging.example.com. See Vercel's guide on preview indexing.
• Netlify adds X-Robots-Tag: noindex to Deploy Previews and older branch deploys, but not to the most recent deploy of each branch, according to its branch deploys documentation.

The leaks happen around those defaults. A DNS record or proxy points the live domain at a preview or deployment-specific URL. Someone adds a noindex rule for staging in next.config.js, netlify.toml, an nginx server block or a CDN response header rule, and the condition that was meant to limit it to staging is wrong or missing. A typical nginx leftover looks like this:

add_header X-Robots-Tag "noindex, nofollow" always;

If you set the header yourself, gate it on the environment, the way Vercel's guide does:

// next.config.js
module.exports = {
  async headers() {
    if (process.env.VERCEL_ENV !== "preview") return [];
    return [{ source: "/:path*", headers: [{ key: "X-Robots-Tag", value: "noindex" }] }];
  },
};

Check it:

curl -sIL https://www.example.com/ | grep -i -E '^(HTTP|x-robots-tag)'

Run it on the exact live hostname, both with and without www, and on a deep page. Headers can be added per path, per host or per file type, so one clean homepage does not clear the whole site. Seeing your page as an AI crawler with curl shows how to repeat the request with crawler user agents.

Fix it: remove the rule or correct its condition, purge the CDN cache, and confirm the header is gone on several URLs.

3. Password protection left on

A staging password is the right way to keep a site private, because it stops people and crawlers alike. Left on after launch, it does the same to everyone. Basic auth returns a 401, and Google states in its HTTP status code documentation that it does not index URLs returning a 4xx status (429 is treated differently) and removes them from the index if they were indexed before.

Some protection layers return 200 with a login form or a coming-soon page instead. That is not better: the crawler gets a page with a password field and none of your content, and that placeholder is all it has to index or summarise.

Check it:

curl -sIL https://www.example.com/ | grep -i '^HTTP'
curl -sL https://www.example.com/ | grep -io '<title>[^<]*'

A 401 or 403, a redirect to a login path, or a title such as "Coming soon" or "Log in" means visitors without a session are not seeing your site. Test from a private window or a machine that has never logged in, since your own session cookie hides the problem.

Fix it: turn off the host's password or deployment protection for the production domain, remove basic auth from the live server block, and deactivate any coming-soon or maintenance plugin.

4. Disallow: / carried over from staging

The staging robots.txt usually reads like this:

User-agent: *
Disallow: /

On a live site it tells every crawler that respects robots.txt, including Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot, that it may fetch nothing. Apple adds a twist: if your file names Googlebot but not Applebot, Applebot follows the Googlebot rules, per Apple's Applebot page.

It also makes the noindex problem worse. Google's block indexing guide says a page must not be blocked by robots.txt for noindex to work, because a blocked crawler never sees the tag, and the page can still appear in results if other pages link to it. Google's robots.txt introduction adds that such a result won't have a description. The robots.txt mistakes guide covers other ways a file blocks AI crawlers by accident.

Check it:

curl -s https://www.example.com/robots.txt

Look for Disallow: / with nothing after the slash in any group that applies to the crawlers you care about. robots.txt applies only to the exact host, protocol and port it is served from, so check www, the bare domain and every subdomain that serves pages.

Fix it: replace the file with your production rules, or delete the physical file if your platform generates a correct one. Then allow for caching: Google generally caches robots.txt for up to 24 hours, OpenAI says its search systems can take about 24 hours to adjust, and Perplexity says changes may take up to 24 hours.

5. Canonical tags pointing at staging

A canonical tag names the URL you want treated as the main version of a page. Google calls rel="canonical" a strong signal in its canonicalization guide, so a live page that says its real address is on staging.example.com sends search engines towards a host that is password protected, noindexed or gone.

<!-- Leftover on the live site -->
<link rel="canonical" href="https://staging.example.com/pricing/">

This usually comes from a site URL setting, an environment variable such as a base URL, or a hard-coded value in a theme. The same value often leaks into Open Graph tags, hreflang, JSON-LD and the sitemap. The sitemaps protocol drops any URL that does not share the sitemap's own protocol and host, so a sitemap full of staging URLs lists nothing useful.

Check it:

curl -s https://www.example.com/pricing/ | grep -io '<link[^>]*canonical[^>]*>'
curl -s https://www.example.com/sitemap.xml | grep -o '<loc>[^<]*' | head -5

Every canonical and every sitemap URL should use the live protocol and hostname. Google recommends absolute canonical URLs and notes that relative ones can cause problems when a testing site gets crawled. The canonical tags guide covers the wider set of canonical conflicts.

Fix it: correct the site URL or base URL setting, search the codebase and database for the staging hostname, regenerate the sitemap, and recheck a few page types.

A launch-day checklist

Run this on the live domain after DNS switches and caches are purged, then again a day later.

1. Status: curl -sIL on the homepage and three deep pages returns 200 after at most one redirect, with no login page.
2. Headers: no x-robots-tag containing noindex or none on HTML pages.
3. Meta robots: no noindex in the HTML of the homepage, a category or listing page, a product or article, and a contact page.
4. CMS: WordPress "Discourage search engines" unticked, SEO plugin defaults reviewed, coming-soon plugins off.
5. robots.txt: production rules on every live host, no blanket Disallow: /, a Sitemap: line pointing at the live domain.
6. Canonicals and sitemap: every URL uses the live protocol and hostname.
7. Staging: still protected by a password, which keeps it out of search without relying on noindex.
8. Search Console: submit the sitemap, use URL Inspection on key pages, and open the robots.txt report to confirm Google fetched the new file.

If AI tools still do not mention you after these pass, the cause is somewhere else, and why ChatGPT can't see your website is the next place to look.

Frequently asked questions

How do I know if my site is set to noindex by mistake?

Fetch a live page with curl -sI and look for an x-robots-tag header, then search its HTML for a robots meta tag. In Google Search Console, URL Inspection reports when indexing is blocked by a noindex rule.

Does noindex stop AI crawlers from fetching my pages?

No. noindex is not a crawl block: crawlers can still fetch the page, and they must fetch it to see the rule. It asks search systems to leave the page out of their index, and only indexed pages are eligible to appear as supporting links in Google's AI Overviews and AI Mode.

How long does it take to recover after removing a leftover noindex?

It depends on how soon crawlers revisit your pages, and no one can promise a date. Requesting indexing in Search Console helps for key URLs. If robots.txt also changed, allow up to about a day: Google generally caches the file for up to 24 hours, and OpenAI and Perplexity give similar timescales.

Should staging use noindex, robots.txt or a password?

A password or other access control is the safest choice, because it keeps out people and crawlers and does not depend on crawlers obeying a rule. Avoid relying on Disallow: / alone: a blocked URL can still appear in Google results if it is linked from elsewhere.

Can a CDN add a noindex header without my knowing?

Yes. A response header rule at the CDN, a server config block or a host's preview defaults can add X-Robots-Tag, and browsers do not show it. Only a header check such as curl -sI reveals it.

Check your live site now

The quickest way to catch all five leftovers is to look at what a crawler receives rather than what your browser shows. The free Crawlito check fetches one public page and reports its status, X-Robots-Tag and meta robots, and the robots.txt rule for each AI crawler, with no account. For a launch or migration, a full audit runs the same checks across up to 500 pages, which matters because a leftover noindex can sit on a single page type.