Shopify Hydrogen robots.txt and sitemap: current and legacy setups
Where Shopify Hydrogen robots.txt and sitemap files live in the 2026-10 starters and legacy Hydrogen, and how Oxygen and Vercel previews treat crawlers.
By Crawlito · 8 min read
In short
- A Hydrogen storefront does not use Shopify's robots.txt.liquid: the app serves /robots.txt from its own route, and Shopify's agent skill notes that Hydrogen ships no helper for it.
- In Hydrogen 2026-10, the React Router template serves robots.txt from app/routes/robots.ts with a 24-hour public cache, and at the time of writing it has no Sitemap line and no sitemap route.
- Vercel Shop, the Next.js 16 starter, serves robots.txt from app/robots.ts and a sharded sitemap index from app/sitemap.xml/route.ts.
- Shopify's legacy docs say Oxygen serves your robots.txt only in the public production environment with a custom domain and replaces it with a disallow-all rule on shared deployments; Vercel previews get X-Robots-Tag: noindex instead.
- Legacy Hydrogen (2026-04 and earlier) used [robots.txt].tsx and [sitemap.xml].tsx routes, and the getSitemap and getSitemapIndex helpers are gone in 2026-10.
On this page
- Which Hydrogen are you running?
- robots.txt in the React Router template on Oxygen
- robots.txt and sitemap in Vercel Shop on Next.js
- Preview deployments: Oxygen versus Vercel
- Legacy Hydrogen: the robots.txt and sitemap routes
- How to check robots.txt, the sitemap and server-rendered JSON-LD
- Frequently asked questions
- Next step
A Shopify Hydrogen storefront serves robots.txt and the sitemap from its own code, not from the theme editor or robots.txt.liquid. In current Hydrogen (version 2026-10), the React Router starter on Oxygen has a robots.ts route and, at the time of writing, no sitemap, while the Next.js starter on Vercel uses Next.js's app/robots.ts and a sitemap route handler. Legacy Hydrogen (2026-04 and earlier) used [robots.txt].tsx and [sitemap.xml].tsx route files. On every version, AI crawlers follow the User-agent: * group unless you add groups that name them.
If your store runs on a Liquid theme instead, the Shopify AI crawlers guide covers robots.txt.liquid. This post is only about headless Hydrogen.
Which Hydrogen are you running?
Shopify's Hydrogen docs now cover version 2026-10 and call 2026-04 and earlier "legacy". The current version is framework-agnostic, with two starter templates:
| Setup | Framework | Host | robots.txt | Sitemap |
|---|---|---|---|---|
| React Router template | React Router 7, framework mode | Oxygen | app/routes/robots.ts | None at the time of writing |
| Vercel Shop | Next.js 16, App Router | Vercel | app/robots.ts | app/sitemap.xml/route.ts plus shards |
| Legacy Hydrogen | Remix, later React Router | Oxygen | app/routes/[robots.txt].tsx | [sitemap.xml].tsx and sitemap.$type.$page[.xml].tsx |
Nuxt, Astro, SvelteKit and SolidStart have development examples in the Hydrogen repository but no starter. The overview page says SEO is the app's job: meta tags, Open Graph and structured data are rendered with your framework's own head or metadata API.
Two consequences matter for crawlers. First, there is no Shopify-generated default to fall back on. Shopify's own hydrogen-robots-txt skill in the Hydrogen repository says Hydrogen ships no helper for robots.txt, and without a route /robots.txt falls through to the framework's 404 page. Under RFC 9309, a 4xx response for robots.txt means crawlers may access any URL on the site, including carts and every sort and filter URL. Second, the hosting platform adds its own rules for non-production URLs, and Oxygen and Vercel do this differently.
robots.txt in the React Router template on Oxygen
The React Router template in the Hydrogen repository registers the route explicitly in app/routes.ts:
route("robots.txt", "routes/robots.ts"),The route is a resource route that returns plain text with Cache-Control: public, max-age=86400. At the time of writing, its rules are:
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /account
Disallow: /checkout
Disallow: /buy/
Disallow: /api/
Disallow: /__shopify
Disallow: /search
Disallow: /collections/*sort_by*
Disallow: /collections/*filter*&*filter*
# Google AdsBot ignores the "*" group and follows only groups that name it.
User-agent: adsbot-google
Disallow: /cart
Disallow: /account
Disallow: /checkout
Disallow: /searchProduct and collection pages stay open (the template has no page or blog routes yet). The skill explains the less obvious lines: /api/ covers the Storefront API proxy and the agent endpoints /api/mcp and /api/ucp/mcp, while /.well-known/ must stay reachable because agents find the UCP business profile there. The filter rule blocks only URLs with two or more filter. parameters, so single-filter collection pages can still work as landing pages.
No AI crawler is named, so GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot all follow the * group. To opt out of training while staying reachable for AI search, add groups to the template string. A crawler obeys only the most specific group that matches it, so a named group has to repeat the paths you still want blocked:
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin
Disallow: /cart
Disallow: /account
Disallow: /checkout
Disallow: /buy/
Disallow: /api/
Disallow: /__shopify
Disallow: /search
Disallow: /collections/*sort_by*
Disallow: /collections/*filter*&*filter*The reasoning behind that split is in allow AI search, block AI training. For the routing details of resource routes, loaders and streaming in React Router 7, see the React Router v7 SEO guide.
The missing sitemap
The template has no Sitemap: line, and its route list has no sitemap route. The skill says to add a Sitemap: line only when the app serves one, built as an absolute URL from the request origin, because a line that points at a 404 shows up as an error in search consoles. The examples/hydrogen app in the same repository has a sitemap library and route files you can adapt. If you migrate from legacy Hydrogen, the sitemap notes in Shopify's migrate-to-hydrogen-2026-10 skill (same repository) say getSitemap and getSitemapIndex are gone in 2026-10 and should be replaced by a small local app/lib/sitemap.ts that runs the same queries.
Watch for cookies on the robots route
The skill warns that when the Hydrogen request context applies response headers, it switches the response to private caching if it carries Set-Cookie or the request used the customer session or Customer Account API. Keep session code out of the robots route, and check the response headers after you deploy.
robots.txt and sitemap in Vercel Shop on Next.js
Vercel Shop uses Next.js's metadata files. At the time of writing, app/robots.ts returns one * group that allows /, disallows cart, account, login and search paths plus sorted and filtered collection URLs, and adds Sitemap: <site url>/sitemap.xml. Each path rule also appears with a /*/ prefix for locale paths, such as /*/cart and /*/search.
Those prefixed rules are worth a second look. Shopify's skill points out that * matches across slashes, so /*/cart also blocks /products/cart-organizer. If your catalog has product or collection handles that begin with cart, account, login or search, list your real locale prefixes instead (/en-ca/cart). The same skill notes that Next.js sends app/robots.ts with Cache-Control: public, max-age=0, must-revalidate; for a longer cache, it suggests a route handler at app/robots.txt/route.ts. The wildcard matching guide shows how these patterns resolve.
The sitemap is a route handler at app/sitemap.xml/route.ts. It asks the Storefront API how many sitemap pages exist for products, collections, pages, blogs and articles, then returns a sitemap index of child files such as /sitemap/products-1.xml, served by app/sitemap/[shard]/route.ts. Because these are your files, you edit them like any other route. The general Next.js points, such as metadata and server rendering, are in the Next.js AI search readiness guide.
Preview deployments: Oxygen versus Vercel
This is where the two hosts differ most, and where staging URLs end up in search results.
Oxygen. Shopify's legacy SEO page says Oxygen serves your robots.txt only in the public production environment, and only when a custom domain is set. For non-production deployments shared through a shareable link or auth bypass token, Oxygen replaces it with a rule that disallows all crawlers. The deployments page adds that deployments are private by default and need a staff login. At the time of writing, the 2026-10 Oxygen pages do not restate this behaviour, so confirm it on your own preview URL rather than assume it.
Vercel. None of that applies to Vercel Shop, because it does not run on Oxygen. Vercel's environments documentation says it adds X-Robots-Tag: noindex to preview deployments and to production deployments opened through their unique URL. It does not add the header when a preview is served from a custom domain assigned to a Git branch, such as staging.example.com. Those domains can be indexed unless you protect them or set the header yourself. New projects also start with Vercel Authentication on previews, unless the team has set a different default.
| Oxygen (per Shopify's legacy docs) | Vercel | |
|---|---|---|
| Default preview access | Staff login required | Vercel Authentication on new projects |
| Shared preview | robots.txt replaced with disallow-all | X-Robots-Tag: noindex |
| Branch staging domain | Not documented | No automatic header |
| Your robots.txt served | Public production with a custom domain | Vercel documents no override of the app's file |
The difference in mechanism matters. A disallow rule stops crawling but does not remove a URL that is already known, while noindex only works if the crawler can fetch the page. The staging noindex leaks guide covers how a header or rule from staging ends up in production, which is the opposite failure.
Legacy Hydrogen: the robots.txt and sitemap routes
On 2026-04 and earlier, Shopify's SEO page says the base template (the skeleton template in the repository) includes these routes:
app/routes/[robots.txt].tsxapp/routes/[sitemap.xml].tsxapp/routes/sitemap.$type.$page[.xml].tsx
If they are missing, the CLI generates them:
npx shopify hydrogen generate route robots
npx shopify hydrogen generate route sitemapBoth files are cached for 24 hours by default. Shopify says added or removed pages appear in the sitemap within a day, and unpublished products drop out automatically. Meta tags come from getSeoMeta in @shopify/hydrogen, which also supports JSON-LD and strips query parameters from canonical URLs by default.
The legacy robots route queried the shop ID to build /{shopId}/checkouts and /{shopId}/orders rules. Shopify's 2026-10 migration skill says to keep your existing rules, remove that query, drop rules for paths the new app no longer serves, and add /admin, /checkout, /buy/, /api/ and /__shopify if they are not already covered. Edit the robots file directly: it is plain text in a template string, so adding AI crawler groups works the same way as in the current template.
How to check robots.txt, the sitemap and server-rendered JSON-LD
Run these against production and against one preview URL.
- robots.txt status and headers. You want
200,text/plain, a publicCache-Controland noSet-Cookie:
curl -sD - https://your-store.com/robots.txt -o /dev/null
curl -s https://your-store.com/robots.txt- The sitemap. If robots.txt has a
Sitemap:line, that URL must return XML, not the app's 404 page:
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://your-store.com/sitemap.xml- Preview protection. On Vercel, look for the header. On Oxygen, open the shared link's robots.txt:
curl -sI https://your-preview-url/ | grep -i x-robots-tag
curl -s "https://your-oxygen-preview-url/robots.txt"- Server-rendered content and JSON-LD. Crawlers that do not run JavaScript only see the first HTML response, so check it for the product name and structured data:
curl -s https://your-store.com/products/linen-shirt | grep -c 'application/ld+json'
curl -s https://your-store.com/products/linen-shirt | grep -o '<title>[^<]*</title>'Check this on the current React Router template in particular. At the time of writing, its product route sets a generic title and description and outputs no Product JSON-LD, while its collection route outputs breadcrumb JSON-LD. Vercel Shop includes a Product schema component with Offer or AggregateOffer. If you started from the React Router template, adding product metadata and Product schema is your job.
Frequently asked questions
Does robots.txt.liquid work on Hydrogen?
No. robots.txt.liquid is a theme template for the Online Store. A Hydrogen storefront serves /robots.txt from a route in its own code.
Does the Hydrogen 2026-10 React Router template include a sitemap?
Not at the time of writing. The template has a robots.txt route but no sitemap route, and the getSitemap helpers from legacy Hydrogen are gone, so you add a local sitemap module and route.
Do Hydrogen templates block AI crawlers?
No. Neither starter names any AI crawler, so they follow the User-agent: * group, which allows products and collections. Add named groups if you want different rules for training crawlers.
Are Vercel preview URLs of a Hydrogen store indexed?
Vercel adds X-Robots-Tag: noindex to preview deployments by default. A preview served on a custom branch domain gets no header, so protect that domain or set the header yourself.
Next step
After you change the robots route or the sitemap, deploy to production and run the free check on one product page to see whether the main AI crawlers can fetch it and what the server-rendered HTML contains. If you are moving a store from legacy Hydrogen to 2026-10, an audit covers up to 500 pages with evidence and prioritized fixes.
Filed under Platform guides. Topics: shopify, robots-txt, react, ai-crawlers, ecommerce, sitemaps