Blog on a subdomain or subdirectory: what changes for crawlers
Blog at blog.example.com or example.com/blog? What Google says, and the practical differences for robots.txt, Search Console, analytics and AI crawlers.
By Crawlito · 6 min read
In short
- Google's SEO Starter Guide says to do whatever makes sense for your business when choosing between subdomains and subdirectories.
- Google's site diversity system generally treats a subdomain as part of its root domain, while site names are supported for domains and subdomains but not subdirectories.
- A subdomain is a separate host with its own robots.txt, so AI crawler rules on the main site do not apply to blog.example.com.
- A Search Console Domain property covers every subdomain, but a URL-prefix property covers only the exact host and path it names.
- Blogs on subdomains often run on a different platform or CDN zone, which brings separate bot protection and crawler defaults that need checking.
On this page
Google does not prefer one over the other. Its SEO Starter Guide says that when choosing between subdomains and subdirectories you should "do whatever makes sense for your business". The differences that matter in practice are operational: a subdomain such as blog.example.com is a separate host with its own robots.txt, often its own platform, CDN settings and bot protection, and it needs the right Search Console property and analytics setup. A subdirectory such as example.com/blog shares all of that with the main site, for better or worse.
If you already have one setup, you rarely need to move. What you need is to know which rules apply to your blog and to check that AI crawlers and search engines can reach it.
What Google says about subdomains and subdirectories
Three Google documents cover the question, and none of them states a ranking preference.
- SEO Starter Guide. Google lists subdomains versus subdirectories among things not to worry about. Its starter guide says it might be easier to manage a site segmented by subdirectories, but other times it might make sense to partition topics into subdomains, depending on your topic or industry.
- Site diversity. Google's ranking systems guide says the site diversity system, which limits how many results from one site appear at the top, "generally treats subdomains as part of a root domain", though sometimes subdomains are treated as separate sites when deemed relevant.
- Site names. Google's site names documentation says site names can appear for domain-level and subdomain-level sites, and are not supported for subdirectories. A blog on a subdomain can therefore show its own site name in results. A blog in a subdirectory shares the main site's.
The AI search operators do not address the question at all at the time of writing. OpenAI, Anthropic and Perplexity document their crawlers per URL and robots.txt per host, not any preference for site structure.
The practical differences at a glance
| Area | Subdirectory (example.com/blog) | Subdomain (blog.example.com) |
|---|---|---|
| robots.txt | Shares example.com/robots.txt | Needs its own blog.example.com/robots.txt |
| Hosting | Same server, or a reverse proxy to another platform | Often a separate platform, pointed at by DNS |
| CDN and WAF rules | Same zone and rules as the main site | May be a different zone, plan or provider |
| Search Console | Covered by the main site's Domain or URL-prefix property | Covered by a Domain property, or needs its own URL-prefix property |
| Analytics | Same host, no extra setup | Check for self-referrals between hosts |
| Sitemaps | Main site's sitemap or a section sitemap | Its own sitemap, referenced from its own robots.txt |
| Google site name | Shares the main site's | Can have its own |
robots.txt and AI crawler access
This is where subdomain blogs most often go wrong. Under RFC 9309, robots.txt applies to the scheme, host and port it is served from. The file at https://www.example.com/robots.txt says nothing about https://blog.example.com/. Each crawler fetches the blog's own file, and if that file does not exist, it treats the blog as allowed.
Two failure modes follow:
- Policies drift apart. You carefully allow OAI-SearchBot and PerplexityBot and opt out of GPTBot on the main site, but the blog platform serves its own default robots.txt with different rules, or none.
- The blog platform blocks crawlers you want. Hosted blog platforms have their own bot settings. Some offer an AI crawler toggle, and some sit behind bot protection you do not control.
With a subdirectory, the opposite risk applies. One robots.txt covers both, so a Disallow meant for the app can catch the blog. If you serve the blog from another platform through a reverse proxy, the blog platform's robots.txt is irrelevant: crawlers read the main site's file. robots.txt across subdomains, protocols and ports covers the matching rules in more depth.
How to check it
curl -s https://www.example.com/robots.txt
curl -s https://blog.example.com/robots.txtRead both. Then confirm the blog answers crawlers with real content, not a challenge page, by repeating a request with a crawler user agent as shown in seeing your page as an AI crawler.
CDN, WAF and bot protection
A subdomain blog is frequently hosted somewhere else: HubSpot, Ghost, Webflow, WordPress.com or a separate server. That means a different set of bot rules. Things to check:
- Is the subdomain on your CDN at all? If DNS points straight at the blog platform, your carefully tuned WAF rules and crawler allowlists do not apply to it.
- If it is on the same CDN account, which zone and rules match it? Rules scoped to
www.example.comby hostname will not matchblog.example.com. - Does the platform add its own protection? Challenge pages from a platform you do not manage are hard to diagnose. WAF challenge pages and AI crawlers shows what they look like in a response.
With a reverse-proxied subdirectory, your CDN sees the blog traffic, but check that caching and security rules written for the app do not interfere with blog pages, and that the proxied platform does not emit canonical tags pointing back at its own hostname.
Search Console and analytics
In Search Console, a Domain property covers all subdomains and both protocols. A URL-prefix property for https://www.example.com/ does not include blog.example.com, so the blog's indexing problems are invisible there. Search Console domain property and missing subdomains explains how to tell which you have.
In Google Analytics 4, moving between subdomains can create self-referrals if the hosts do not share a cookie domain. Google's cross-domain measurement help recommends setting it up for subdomains to avoid them. If your blog and app are on different domains entirely, see GA4 cross-domain tracking from blog to app.
Which to choose for a new blog
If you are starting fresh, let operations decide:
- Choose a subdirectory when one team runs the site and you want one robots.txt, one set of edge rules and one analytics setup. It is the option with fewer places for crawler rules to drift apart.
- Choose a subdomain when the blog must run on a separate platform you cannot proxy, when a different team owns it, or when you want it to have its own site name in Google.
Either way, write down who owns the blog's robots.txt, DNS and CDN settings. Most crawler access problems on blogs come from nobody owning them.
Moving a blog between a subdomain and a subdirectory
If you do decide to move, treat it as a site migration:
- Map every old URL to its new URL, one to one.
- Serve
301redirects from the old host to the new paths, and keep them in place for a long time. - Update canonical tags, internal links and sitemaps to the new URLs before launch.
- Make sure the new location's robots.txt and edge rules allow the crawlers you want, before the redirects go live.
- Add or update the Search Console property, and watch indexing for both old and new URLs.
Website migrations and AI search covers the steps and checks in detail. A move is not a documented way to gain visibility, so only do it if it solves a real operational problem.
Frequently asked questions
Is a subdirectory better for SEO than a subdomain?
Google does not say so. Its SEO Starter Guide says to do whatever makes sense for your business, and its site diversity system generally treats subdomains as part of the root domain.
Does my main robots.txt cover my blog subdomain?
No. robots.txt applies only to the host it is served from. blog.example.com needs its own robots.txt with the rules you want for AI crawlers and search engines.
Do AI crawlers treat subdomains differently?
They fetch and obey robots.txt per host, like search engines. OpenAI, Anthropic and Perplexity do not document any other difference between subdomains and subdirectories.
Should I move my blog from a subdomain to a subdirectory?
Only if it solves a concrete problem, such as duplicated tooling or inconsistent rules. A move carries migration risk, and Google does not document a benefit from the structure itself.
Next step
Fetch both robots.txt files and test one blog post and one main-site page for crawler access. The free check runs that test for a single URL with no account, and an audit can cover the blog and main site together.
Filed under Strategy and audits. Topics: robots-txt, search-console, analytics, ai-crawlers, cdn, google