Does robots.txt apply to subdomains? Hosts, protocols and ports
robots.txt only covers the exact host, protocol and port it is served from. How that works for www vs apex, subdomains, CDN and asset hosts, and staging.
By Crawlito · 7 min read
In short
- A robots.txt file only applies to the exact scheme, host and port it is served from, so www.example.com and example.com each need their own answer.
- Google's specification says a robots.txt on a subdomain is only valid for that subdomain and crawlers do not look for robots.txt files in subdirectories.
- When robots.txt redirects, RFC 9309 says the rules are followed in the context of the initial host, so an apex-to-www redirect of the file is a valid setup.
- CDN, image and API hosts such as cdn.example.com answer robots.txt requests for themselves, and a Disallow there can stop crawlers from fetching assets your pages use.
- Staging hosts need their own robots.txt or authentication, and a staging file copied to production is a common way to block the live site.
On this page
No, a robots.txt file does not cover subdomains. It applies only to the exact combination of protocol, host name and port it is served from. The file at https://www.example.com/robots.txt says nothing about https://example.com/, https://blog.example.com/ or http://www.example.com:8080/. Each of those is a separate site to a crawler, and each is checked for its own /robots.txt.
This catches people out in both directions: a block they thought covered everything leaves a subdomain open, or a forgotten file on an asset or staging host blocks something they wanted crawled. Below is what the standard and Google's specification say, and how to check every host your site actually uses.
What "per host, protocol and port" means
RFC 9309, the Robots Exclusion Protocol standard, says the rules must be in a file named /robots.txt in the top-level path of the service, written as scheme:[//authority]/robots.txt. The authority is the host name plus an optional port. So the file belongs to one scheme and one authority.
Google's robots.txt specification spells out the consequences in its "file location and range of validity" section:
- A robots.txt on a subdomain is only valid for that subdomain.
- A file at
https://example.com/robots.txtis not valid for other subdomains, protocols or port numbers. - A file on a non-standard port is only valid for content served through that port.
- Crawlers do not look for robots.txt files in subdirectories.
/blog/robots.txtis just a file. - A robots.txt served on an IP address as the host name applies only to that IP address as a host, not to every site hosted on it.
Here is how that plays out for one organisation:
| robots.txt URL | Covers | Does not cover |
|---|---|---|
https://www.example.com/robots.txt | https://www.example.com/... | https://example.com/, https://shop.example.com/ |
https://example.com/robots.txt | https://example.com/... | https://www.example.com/ |
https://blog.example.com/robots.txt | https://blog.example.com/... | https://example.com/blog/ |
http://www.example.com/robots.txt | http://www.example.com/... | https://www.example.com/ |
https://www.example.com:8443/robots.txt | https://www.example.com:8443/... | https://www.example.com/ (port 443) |
https://www.example.com/blog/robots.txt | Nothing: crawlers do not read it | Everything |
Treat AI crawlers the same way. OpenAI, Anthropic and Perplexity document their crawlers' tokens for robots.txt, and none of them documents reading one host's file to decide about another host. Our robots.txt guide for AI crawlers covers what to put in each file.
www vs apex: which file counts?
Most sites serve pages on one host and redirect the other. Say example.com redirects everything to www.example.com. A crawler that requests https://example.com/robots.txt gets a redirect to https://www.example.com/robots.txt.
RFC 9309 covers this case. Crawlers should follow at least five consecutive redirects, even across hosts, and if they reach a robots.txt file that way, they must parse it and follow its rules in the context of the initial authority. So the www file is applied to the apex host. Google's specification says the same about redirects: it follows at least five hops, and if it finds no file after that, it treats robots.txt as a 404.
That makes three setups safe:
- Redirect the file along with everything else. The apex
/robots.txtredirects to the www file, which then governs both hosts. - Serve an identical file on both hosts. Fine, as long as you update both together.
- Serve a short file on the redirecting host. The apex host only ever returns redirects, so a permissive file there is harmless.
The setup to avoid is a different, stale file on the host that does not redirect. For example, a migration moved the site to www, but the apex host still serves an old Disallow: / from the previous platform, and some links still point at apex URLs.
Two things a redirect does not do: it does not let JavaScript or meta refresh count as a redirect (Google says it does not follow those for robots.txt), and it does not merge two files. If both hosts serve their own file, each host gets its own rules.
Subdomains: blog, shop, docs and help centers
Every subdomain answers for itself. That is often what you want, because a help center on a hosted platform at help.example.com and a store at shop.example.com are run by different tools. It also means:
- Blocking AI training on
wwwdoes not block it ondocs. If your documentation host has no robots.txt, or a platform default, crawlers follow that file, not yours. - A hosted platform may serve its own robots.txt. Help desks, status pages, landing page builders and shops on a custom subdomain often generate the file for you. Read what each one actually serves.
- A subdirectory is not a subdomain.
example.com/blog/is covered byexample.com/robots.txt. If you move a blog fromblog.example.comto/blog/, its rules have to move into the main file. Our post on subdomain vs subdirectory blogs covers the other effects of that choice.
Search Console has a related trap: a URL-prefix property for www does not report on other subdomains, so check each host you run, or use a Domain property.
CDN, image and asset hosts
Static files often come from another host: cdn.example.com, static.example.com, assets.example-cdn.net or a cloud storage bucket. Crawlers request /robots.txt on those hosts too, before fetching an image, stylesheet or script from them.
Two problems show up here:
- An asset host blocks everything. Some teams add
Disallow: /to a CDN host to stop its URLs appearing in search. That also stops crawlers that respect robots.txt from fetching CSS, JavaScript and images from it. For a renderer such as Googlebot, missing CSS and JavaScript can change what the page looks like. The guide to blocked CSS and JavaScript shows how to spot this. - An asset host has no robots.txt at all and returns an error page. RFC 9309 treats a
4xxresponse as "unavailable", which allows crawling. A5xxis "unreachable", and the crawler must assume complete disallow. A storage bucket that returns500or a timeout for/robots.txtcan therefore stop asset fetches. The post on what crawlers do when robots.txt returns a 404 or 5xx goes through each status.
A safe default for an asset host you want crawlers to use is a file that allows everything and returns 200:
User-agent: *
Allow: /If you want image URLs on the CDN to stay out of search results, use X-Robots-Tag: noindex headers on those responses instead of a robots.txt block, so the files can still be fetched.
Staging, preview and port-based hosts
Staging environments are separate hosts, so they need their own protection: staging.example.com, preview-123.example.vercel.app or www.example.com:8080. Two rules of thumb:
- Protect staging with authentication, not only robots.txt. A
Disallow: /keeps compliant crawlers out, but it is a public file that anyone can read, and it enforces nothing. A password or IP restriction keeps everyone out. - Never deploy the staging robots.txt to production. The most common robots.txt incident on a launch is a
Disallow: /that was correct on staging and got copied to the live host. Our staging noindex leaks guide has a launch checklist for it.
Port-based environments are their own case. A file on port 8080 only covers port 8080, so a dev server reachable on a public port needs its own rules or, better, no public access.
How to check every host your site uses
List the hosts first, then request /robots.txt on each one. The list usually includes the apex, www, every subdomain in your sitemap or navigation, and every host your pages load assets from.
for host in example.com www.example.com blog.example.com shop.example.com cdn.example.com; do
printf "%-22s " "$host"
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" "https://$host/robots.txt"
doneThen read the response for each host:
200with a file: read it and check the groups for the crawlers you care about. The robots.txt testing guide has a routine per crawler.301or308to another host: follow it and confirm the target file is the one you want applied to this host.404or410: no rules apply on this host. That is fine if you intend it.5xxor a timeout: fix this first. Compliant crawlers treat the host as fully disallowed while it lasts.
To find asset hosts, pull them from a page's HTML:
curl -s https://www.example.com/ | grep -oE '(src|href)="https?://[^/"]+' | sed -E 's#.*//##' | sort -uRepeat the check for http:// on hosts that still answer on plain HTTP, and for any non-standard port that is publicly reachable.
Frequently asked questions
Does robots.txt on my main domain block subdomains?
No. Each subdomain is checked for its own /robots.txt. Google's specification says a file on a subdomain is only valid for that subdomain, and the same is true in reverse.
Do I need separate robots.txt files for http and https?
Yes in principle, because the protocol is part of what the file covers. In practice most sites redirect all HTTP requests to HTTPS, including /robots.txt, and the redirected file is then applied to the original host.
Can I put a robots.txt file in a subfolder?
You can, but crawlers will not read it. Google says crawlers do not look for robots.txt files in subdirectories, and RFC 9309 places the file only at the top-level path.
What happens if my CDN host has no robots.txt?
If it returns 404, crawlers treat it as having no rules and fetch freely. If it returns a 5xx error or times out, RFC 9309 says crawlers must assume complete disallow, which can stop them fetching your assets.
Should staging have a robots.txt with Disallow: /?
It does no harm, but rely on authentication to keep staging private. And make sure that file never reaches production.
Check the hosts your pages depend on
A robots.txt problem on a host you forgot about looks exactly like a crawler ignoring you. Start with the free single-page check, which reads the robots.txt for the host you test. For a site with several subdomains and asset hosts, an audit reviews each host and lists the fixes in order.
Filed under Access and robots.txt. Topics: robots-txt, ai-crawlers, cdn, troubleshooting