Astro SEO for AI crawlers: islands, client:only and sitemaps
Astro serves static HTML by default, which suits AI crawlers. Where client:only, server islands, SSR routes and missing site config still hide content.
By Crawlito · 6 min read
In short
- Astro prerenders every page to HTML by default, and framework components without a client directive render as HTML with no JavaScript, so most Astro content is visible to crawlers that do not run scripts.
- A component with client:only skips server rendering entirely, so crawlers that read raw HTML see only its fallback slot, or nothing.
- Server islands (server:defer) ship a placeholder and a script in the initial HTML and fetch the real content afterwards, so their content is missing from the raw HTML.
- The @astrojs/sitemap integration requires site in astro.config and cannot list dynamic routes rendered on demand.
- On-demand pages must set Astro.response.status or return a Response for missing content, or a missing product can answer 200.
On this page
Astro is one of the friendliest frameworks for AI crawlers, because it renders pages to plain HTML by default and only adds JavaScript where you ask for it. Content in .astro components, and framework components without a client:* directive, is in the HTML a crawler downloads. The problems come from four specific features: client:only components, server islands, routes rendered on demand, and integrations that need configuration such as the site URL. Check those and an Astro site is usually in good shape.
This matters because many AI crawlers read the HTML your server sends and do not run JavaScript, as covered in do AI crawlers run JavaScript. Anything Astro only renders in the browser is invisible to them.
How Astro renders pages: static by default, on demand when asked
Astro's on-demand rendering guide says the whole site is prerendered by default. That is the 'static' output mode: every page becomes an HTML file at build time.
You can change that in two ways:
- Per page:
export const prerender = falsemakes one page or endpoint render on demand, on each request. - Whole site:
output: 'server'inastro.config.mjsmakes on-demand rendering the default, andexport const prerender = trueopts single pages back into prerendering.
Either way, rendering on demand needs an adapter for your host. Both modes produce HTML on the server, so both are readable by crawlers. The difference that matters for crawlers is not static versus server, but what happens in the browser afterwards.
Islands and client directives: what ends up in the HTML
Astro's directives reference defines how each framework component (React, Vue, Svelte and others) is delivered:
| Usage | Server HTML | JavaScript | Crawler without JS sees |
|---|---|---|---|
<Widget /> (no directive) | Rendered | None | Full content |
<Widget client:load /> | Rendered | Hydrates on page load | Full content |
<Widget client:idle /> | Rendered | Hydrates after initial load | Full content |
<Widget client:visible /> | Rendered | Hydrates when scrolled into view | Full content |
<Widget client:only="react" /> | Skipped | Renders in the browser only | Fallback slot, or nothing |
<Widget server:defer /> | Placeholder | Fetched after load | Fallback slot, or nothing |
The client:load, client:idle and client:visible directives are safe for content: the component is rendered on the server first and only made interactive later. The two rows in bold are where content goes missing.
The client:only pitfall
The docs say client:only "skips HTML server rendering, and renders only on the client". You also have to name the framework, as in client:only="react", because Astro never runs the component on the server and cannot infer it.
People reach for client:only when a component touches window or localStorage during render and breaks the build. The quick fix works, and the component's text disappears from the HTML. If that component holds a pricing table, product specs or an FAQ, AI crawlers that read raw HTML never see it.
Better fixes, in order:
- Make the component safe to render on the server. Move browser-only code into an effect or event handler, then use
client:loadorclient:visible. - Split it. Render the content as plain HTML in an
.astrocomponent, and keep only the interactive control as aclient:onlyisland. - If it must stay client-only, give it a real fallback. A child element with
slot="fallback"is rendered in the HTML until the component loads. Put the essential text there, not a spinner.
---
import PriceCalculator from "../components/PriceCalculator.jsx";
---
<PriceCalculator client:only="react">
<div slot="fallback">
<p>Starter: $29 a month. Team: $99 a month. Billed annually.</p>
</div>
</PriceCalculator>Server islands load their content after the page
server:defer turns a component into a server island. Astro's server islands guide explains the delivery: at build time Astro omits the component and injects a script in its place, and in the browser that script requests the island's HTML from a separate endpoint. The fallback slot is rendered with the page initially and replaced when the content arrives.
That is ideal for personalised bits such as an avatar or a cart count. It is the wrong choice for anything you want crawlers to read, because the raw HTML holds only the fallback.
How to check what crawlers see on an Astro page
Look at the HTML the server sends, not the page in your browser:
curl -s https://www.example.com/pricing | grep -c "Team: \$99"A count of 0 for text you can see in the browser means it is rendered in the browser. Then search your source for the two directives that skip server rendering:
grep -rn "client:only\|server:defer" src/Review each match and ask whether the content inside should be readable by crawlers. Raw vs rendered HTML covers the comparison in more depth.
Three other places hide content in Astro projects more often than people expect:
- Data fetched in the browser. A React or Vue island that calls an API in an effect renders an empty shell on the server, even with
client:load. Fetch the data in the Astro frontmatter instead and pass it to the component as props, so it is in the HTML. - Tabs and accordions built as islands. If only the active tab is rendered on the server, the other panels exist only after a click. Render all panels in the HTML and toggle visibility with CSS or the
hiddenattribute. - Embedded third-party widgets. Reviews, store locators and comparison tables loaded from another service by script are not part of your HTML at all.
If you are deciding how much of a site to prerender, SSR, SSG and CSR for AI search compares the trade-offs.
robots.txt and sitemaps in Astro
Sitemap integration
The @astrojs/sitemap integration is added with npx astro add sitemap. Two limits matter:
- It needs
site. You must setsiteinastro.config.mjsto a full URL starting withhttp://orhttps://, because sitemaps need absolute URLs. - It cannot list on-demand dynamic routes. Dynamic routes built with
getStaticPaths()are included. Routes likesrc/pages/products/[slug].astrorendered on demand are not. If your product or article pages are rendered on demand, generate their sitemap yourself from the same data source.
The build writes sitemap-index.xml, which points to sitemap-0.xml and further numbered files on large sites. Use the filter option to leave out pages such as thank-you or search pages.
// astro.config.mjs
import { defineConfig } from "astro/config";
import sitemap from "@astrojs/sitemap";
export default defineConfig({
site: "https://www.example.com",
integrations: [
sitemap({ filter: (page) => !page.includes("/thank-you/") }),
],
});robots.txt
The simplest option is a static public/robots.txt, which Astro copies as is. The sitemap docs also show an endpoint at src/pages/robots.txt.ts that builds the Sitemap: line from your site value, so the URL is never out of date:
import type { APIRoute } from "astro";
const getRobotsTxt = (sitemapURL: URL) => `User-agent: *
Allow: /
Sitemap: ${sitemapURL.href}
`;
export const GET: APIRoute = ({ site }) => {
const sitemapURL = new URL("sitemap-index.xml", site);
return new Response(getRobotsTxt(sitemapURL));
};Add groups for specific AI crawlers here if you need them. robots.txt for AI crawlers covers which tokens to use.
Status codes, canonicals and other Astro details
- 404 pages. A
src/pages/404.astrofile builds to404.html, which most hosts pick up for unknown URLs. Confirm with curl that your host returns it with a404status, not200. On on-demand pages, a missing record does not trigger it automatically. SetAstro.response.status = 404in the page frontmatter, or returnnew Response(null, { status: 404 }), otherwise a deleted product answers200with an empty template. Astro's docs note that response headers can only be set at the page level, not inside components. - Redirects. In on-demand pages, use
return Astro.redirect("/new-url", 301)for moved content, so crawlers get a permanent redirect rather than a page that redirects with JavaScript. - Canonical tags. Astro does not add them for you. Build them in your layout from
Astro.siteandAstro.url.pathname, and keep thetrailingSlashsetting consistent with the URLs in your sitemap. - View transitions and client routing. These change how users move between pages but not the HTML each URL returns. Check one URL with curl to confirm.
Frequently asked questions
Is Astro good for SEO and AI crawlers?
Its defaults are. Astro prerenders pages to HTML and ships no JavaScript for components without a client directive, so content is in the raw HTML. Problems come from client:only, server islands and missing configuration.
Does client:visible hide content from crawlers?
No. client:visible renders the component on the server and only delays hydration until it scrolls into view. The HTML is already there for crawlers.
Why are my product pages missing from the Astro sitemap?
If they are dynamic routes rendered on demand, the sitemap integration cannot generate entries for them. Prerender them with getStaticPaths() or build a sitemap endpoint from your product data.
Do I need an adapter for robots.txt in Astro?
No. A file in public/robots.txt, or a src/pages/robots.txt.ts endpoint that is prerendered, works on a fully static site.
Next step
Run curl against your most important templates and look for text that only appears in the browser. The free check does a similar test for one page with no account, and a full audit checks rendering, robots.txt and sitemaps across up to 500 pages.
Filed under Platform guides. Topics: javascript, rendering, ssr, sitemaps, robots-txt, raw-html